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Overview 



Formative assessments — assessments that measure what students do and do not know, so that teachers can 
modify their instmction accordingly — have been widely hailed as a potential vehicle for improving student 
achievement. Yet little solid research evidence exists about their effectiveness, especially in reform-rich 
school districts. This study examines the effects of the Formative Assessments of Student Thinking in Read- 
ing (FAST-R) initiative in the Boston Public Schools system (BPS), where the use of data to improve in- 
stmction is a general priority of the school district. The study looks at changes in reading scores over time at 
21 BPS schools that operated FAST-R during the 2005-2006 and 2006-2007 school years and changes at a 
group of comparison schools serving demographically similar students during the same period. 

The Boston Plan for Excellence (BPE), a not-for-profit school reform organization, created and operates 
EAST-R. The study intervention involved administering a series of short student assessments whose items 
resemble those in the Massachusetts Comprehensive Assessment System (MCAS), the state’s “high-stakes” 
assessment used to measure the performance of both schools and students, and focus on students’ reading 
comprehension skills. BPE staff members compiled the results of the assessments into easy-to-use reports 
that contained information about each student. Then a BPE instmctional data coach met with the teachers at 
each school to review the reports and to suggest how teachers could respond to students’ learning needs. 
(One BPE coach served most of the schools, and another BPE coach served the balance.) 

The MDRC evaluation includes process and impact analyses. The process analysis found that teachers at the 
EAST-R schools who took a survey administered as part of the study reported that the professional devel- 
opment they received from the BPE EAST-R coaches was helpful and contributed to their understanding of 
data and their ability to work with students. At the same time, while the intervention was implemented as 
intended (it was meant to be flexible and to provide as much or as little coaching to individual schools as 
administrators and teachers sought), it was not very intensive; the majority of survey respondents spent only 
one to five hours with the EAST-R data coach during the 2006-2007 school year. Moreover, comparison- 
school teachers who took the survey reported receiving at least as much professional development as their 
EAST-R counterparts, were as likely to find it useful, and spent as much or more time analyzing data, in- 
cluding data from other (non-EAST-R) formative assessments. 

The impact analysis examines the effects of EAST-R on the reading test scores of third- and fourth-graders. 
EAST-R’s impacts on student achievement — that is, the difference that EAST-R made over and above 
what was going on in the comparison schools — are generally positive but not statistically significant, as 
measured by MCAS reading scores. In other words, these differences could have arisen by chance. Effects 
on another measure of student reading, the Stanford Achievement Test, are more mixed but are also not sta- 
tistically significant. 

While EAST-R schools put in place a particular model of data utilization, other BPS schools were pursuing 
similar goals, and this fact, along with the intervention’s lack of intensity, may have undercut the likelihood 
that it would generate substantial and statistically significant impacts in this evaluation. Thus, this single 
study in a single district is not the last word on the potential of EAST-R. Much remains to be discovered 
about how teachers can best learn to use data to improve their instmction and boost the achievement of their 
students. 
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Preface 



When the No Child Left Behind Act was passed in 2001, many school districts were al- 
ready looking at and testing the effectiveness of various instructional strategies for building read- 
ing skills, particularly for younger children. Among those strategies, the use of student data to in- 
form teaching — and, in turn, to improve learning — has often been identified as a potentially 
powerful educational tool. One important means of generating such data is the use of “formative 
assessments” — tests or activities that measure student learning and provide feedback to teachers 
that they can use to adapt their teaching practices to meet student needs. Yet formative assess- 
ments have rarely been scrutinized to determine their effectiveness. This report describes an initia- 
tive that was designed to enhance the usefulness of formative assessments and presents an evalua- 
tion of its impact on students’ reading skills in Boston public elementary schools. 

The Formative Assessments of Student Thinking in Reading (FAST-R) program, as the 
initiative is called, was created and is operated by the Boston Plan for Excellence (BPE), a not-for- 
profit school reform organization. BPE created a series of formative assessments that teachers 
could administer to their students at various points during the school year, generating data that 
teachers would then use to inform their reading instmction. Recognizing, however, that merely 
generating data is not enough if teachers don’t know how to interpret and use those data, BPE in- 
cluded a second component in the EAST-R program — professional development provided by an 
“instmctional data coach,” who helped teachers understand the data. Encouraged by positive 
feedback from a qualitative study of EAST-R’s early operations, BPE contracted with MDRC to 
evaluate the initiative’s impacts in 21 schools in the Boston Public Schools system. Conducted 
with support from the William and Flora Hewlett Foundation, the evaluation found that while 
teachers at the FAST-R schools reported, by and large, that the program had been helpful, no im- 
pacts on students’ reading test scores were registered in those schools compared with a group of 
schools serving similar students that did not implement FAST-R. It should be noted, however, that 
the program treatment may not have been intensive enough to have an impact, particularly in an 
environment where many teachers — including those in the comparison schools — were already 
using data and participating in professional development aimed at improving reading instmction. 

The FAST-R evaluation was one test of formative assessments in one school district, and, 
as such, its findings cannot be considered a definitive statement on the effectiveness of training 
teachers in the use of data to guide reading instmction. Given that the use of data to improve learn- 
ing is such an important idea, different tests of this strategy in different settings are very much in 
order. It is MDRC’s hope that this evaluation will help to shape future research and that such re- 
search will yield findings that stimulate educators’ ongoing efforts to help children boost their 
reading skills — whether by proceeding along a proven, evidence-based path or by recognizing 
the need to forge a new one. 



Gordon Berlin 
President 
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Executive Summary 



Formative assessments — that is, assessments administered in order to measure what 
students do and do not know, so that teachers can modify their instruction accordingly — have 
been widely hailed as a potential vehicle for improving student achievement. To date, however, 
little rigorous research has been done on the impacts of formative assessment and its resulting 
data-driven instmction, particularly in reform-rich urban school districts. This study, funded by 
the William and Flora Hewlett Foundation, is a step toward filling that knowledge gap. It 
examines the effects of the Formative Assessments of Student Thinking in Reading (FAST-R) 
initiative as it operated in 21 schools in the Boston Public Schools system (BPS) during the 
2005-2006 and 2006-2007 school years. 

The Boston Plan for Excellence (BPE) — a not-for-profit organization that works with 
the BPS central office and individual schools to design, pilot-test, and implement new reforms 
aimed at improving teaching and learning — created and operates EAST-R. The intervention 
(as operated during the study period) involved a series of short assessments whose items 
resemble the multiple-choice questions contained in the Massachusetts Comprehensive As- 
sessment System (MCAS), the state’s “high-stakes” assessment used to measure both student 
and school performance, and administered annually to students in grades 3 through 8 and grade 
10. EAST-R questions, like those on the MCAS, focus on two essential reading skills: the ability 
to find evidence in the text that supports an explicit point and the ability to make inferences from 
the available information to support a valid interpretation. 

Schools are free to choose those EAST-R assessments that comport best with their les- 
son plans and to administer them on a schedule that best suits their needs. After students have 
taken an assessment, BPE staff score their answer sheets and compile the results in reports that 
are designed to be easy to use and that contain information about how each student, as well as 
groups of students, scored on each assessment item. The reports are meant to help teachers 
understand not only how many students came up with the right answers but also what mistakes 
in reading or reasoning led students to come up with the wrong ones. Then a BPE instructional 
data coach meets with the teachers at each school to review the reports, help them learn how to 
interpret the data, and suggest how they can respond to students’ learning needs. During the 
period under study, one BPE coach worked with most of the EAST-R elementary schools. 
(Another coach worked with the rest of the schools.) 

The use of data to inform instruction is a general priority in the Boston Public Schools 
district, not just in those schools that have elected to implement EAST-R. In fact, BPS has 
included the use of student data to identify student needs, improve instmction, and assess 
progress as one of “The Six Essentials” (guiding principles) of its “Whole-School Improve- 
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ment” model, which is used throughout the Boston Public Schools system.' (Other “essentials” 
include a focus on literacy and mathematics instruction and professional development for 
principals and teachers that is focused on improving instmctional skills.) FAST-R was intended 
to complement the district’s own professional development supports, and, while FAST-R 
schools used their own unique assessment tool and put in place a particular model of data 
utilization, other Boston public schools were pursuing similar goals, sometimes through broadly 
similar means. For instance, teachers at non-FAST-R schools had access to other formative 
assessments, worked with literacy coaches to improve their instmctional techniques, and could 
participate in various other types of professional development. This fact needs to be borne in 
mind when considering the evaluation findings, since outcomes at the FAST-R schools are 
compared with outcomes at these other schools. 



The Evaluation Design 

The evaluation includes a process analysis and an impact analysis. The process analysis 
was intended to provide information about how teachers in the FAST-R schools used what they 
had learned and, more generally, about how they regarded the initiative. Equally if not more 
important, it aimed to shed light on the professional development efforts that took place both in 
the FAST-R schools and in a group of comparison schools that served students who were 
similar to those in the FAST-R schools but did not put the initiative into place. In this way, the 
process analysis helps to establish a context for the impact analysis findings. 

Surveys administered in the spring of 2007 to principals and to th ir d- and fourth-grade 
teachers at both sets of schools are the key source of data for the process analysis. Unfortunate- 
ly, during this period, the Boston Teachers Union was negotiating a new contract. It seems 
likely that many teachers were unwilling to undertake any noninstnictional activities until 
contract issues were resolved; in any case, the response rates for the surveys were low (about 54 
percent for teachers in both the FAST-R and the comparison schools, and even lower for 
principals). Because of the low response rates, and because some schools participating in the 
impact analysis did not supply any survey responses at all, the survey findings can be viewed 
only as suggestive rather than definitive in illuminating the similarities and differences between 
program and comparison schools. 

The impact analysis uses a comparative intermpted time series design to examine the 
effects of FAST-R on the reading test scores of third- and/or fourth-graders in 21 schools that 
implemented the program during the 2005-2006 and 2006-2007 school years. Attention centers 
on these grade levels because they are the earliest grades in which the MCAS is routinely 



' As of fall 2008, BPS was using “The Seven Essentials” in its Whole-School Improvement Model. 
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administered and because in these grades students face new challenges as reading comprehen- 
sion replaces “decoding” (sounding out words) as the major focus of instmction. The FAST-R 
schools selected for the study were ones identified by BPE as having actively implemented the 
intervention during the study period. 

The impact analysis includes two comparisons. The first comparison is between test 
score outcomes at the 21 FAST-R schools before and after the initiative was put in place. To 
draw this comparison, scores on each outcome over a five-year pre-intervention “baseline” 
period were used to create a trend line and to project that trend line into the post-implementation 
period. The difference between the actual and projected scores during this post-implementation 
period represents the “deviation from trend” for the FAST-R schools. 

If one were to look at the program schools alone, however, it would be impossible to 
determine how much of the observed change was attributable to FAST-R and how much that 
change reflected other developments throughout BPS. A second comparison, therefore, involves 
measuring changes in outcomes over time at a set of BPS schools that did not implement 
FAST-R but whose student populations resemble those at the FAST-R schools in terms of 
demographic characteristics and prior achievement. Thirty-six schools were selected as compar- 
ison schools. As with the FAST-R schools, baseline scores were used to project a trend line for 
the comparison schools, and the difference between projected and actual scores represents the 
deviation from trend for these schools. The impact of FAST-R is the difference between the 
deviation from the trend for the program schools and the deviation from the trend for the 
comparison schools. 

The impact analysis relies on individual student records obtained from BPS. Students’ 
general reading achievement is measured using three outcomes from two standardized tests: the 
average reading comprehension total score on the MCAS; the percentage of students scoring at 
or above the “proficient” level on the MCAS; and the average reading total score on the 
Stanford Achievement Test, version 9 (SAT-9). (Because the test items in the FAST-R assess- 
ment are so similar to those on the MCAS, use of a second test provides reassurance that any 
positive impacts on the MCAS are not simply the result of increased student familiarity with 
that assessment.) In addition to these general measures, the analysis measures how well students 
in program and comparison schools performed in answering MCAS questions designed to 
measure students’ specific reading skills in terms of finding evidence in and making inferences 
from text. 
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Findings on FAST-R and Other Professional Development 
Activities in the Boston Public Schools System 

• Teachers at the FAST-R schools who responded to the survey reported 
that the professional development they received from the BPE FAST-R 
coaches was helpful and contributed to their understanding of data and 
their ability to work with students. 

While, as noted above, the survey respondents were not necessarily typical of all teach- 
ers at the FAST-R schools, their views of FAST-R were notably positive. They reported that the 
initiative had helped them to understand students’ thinking and to use student data in reflecting 
on their instructional practices. 

• The majority of FAST-R teachers reported spending a limited amount 
of time with the BPE coach. 

Just over 60 percent of the FAST-R survey respondents reported spending one to five 
hours with their FAST-R data coach over the course of the 2006-2007 school year. Only 13 
percent reported spending 1 1 or more hours with the coach. This is not surprising, given the 
many demands on the time of the FAST-R coach who worked with elementary schools. 

• Compared with their counterparts at the FAST-R schools, teachers res- 
ponding to the survey at the comparison schools reported participating 
in at least as much professional development, were as likely to find that 
professional development useful, and spent as much or more time ana- 
lyzing student data, including data from formative assessments, 

FAST-R may have contributed to teachers’ knowledge and understanding, but that con- 
tribution was not unique. Teachers at the comparison schools spent as much time as did teachers 
at the FAST-R schools engaging in Collaborative Coaching and Learning — the district’s 
school-based professional development model, which emphasizes collaboration among teachers 
— or working with literacy coaches, as well as observing other teachers’ classrooms; compari- 
son-school teachers actually reported spending more time on curriculum-specific professional 
development. Like the FAST-R teachers, they believed that the professional development was 
helpful for conducting their reading classes, leading discussions, creating assignments, and 
placing students in groups according to reading level. They also reported spending as many 
hours as the FAST-R teachers in analyzing formative assessments, and they spent substantially 
more hours analyzing the previous year’s MCAS results. 
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Findings on FAST-R’s Impacts on Students’ Reading Skills 

• FAST-R unfolded in an environment of stable or improving reading 
scores. 

During the five years preceding FAST-R’s implementation, average reading scores on 
the MCAS and SAT-9 held steady for third-graders and improved for fourth-graders at both the 
FAST-R and the comparison schools. 

• FAST-R’s impacts on student achievement — that is, the difference that 
FAST-R made over and above what was going on in the comparison 
schools — are generally positive but not statistically significant, as 
measured by MCAS reading scores. In other words, these differences 
could have arisen by chance. Effects on the SAT-9 reading scores are 
more mixed, but also not statistically significant. 

The achievement gains of students at the FAST-R schools during the follow-up period 
were somewhat larger than those of students at the comparison schools, as measured by the 
MCAS, but the differences are not statistically significant. This was the case for both third- and 
fourth-graders. Average scores for the two groups did not differ, nor did the percentage of 
students whose scores placed them in the “proficient” or “advanced” categories. FAST-R was 
associated with both positive and negative impacts on SAT-9 scores, but none of these impacts 
is statistically significant. 

• The initiative did not have an impact on students’ ability to find evi- 
dence in or make inferences from text, as measured on the MCAS, 

Strengthening students’ reading comprehension skills as measured by their ability to 
find evidence in and make inferences from text was the key objective of FAST-R. The study 
specifically examined students’ responses to MCAS questions that were designed to tap into 
these abilities. The results indicate that students at the FAST-R and comparison schools made 
similar progress in their ability to answer these two kinds of questions. 

• FAST-R did not produce consistent impacts for particular subgroups of 
students or schools. 

Although FAST-R had a positive and statistically significant effect on the percentage of 
fourth-grade boys who scored at the “proficient” or “advanced” level on the MCAS, the 
difference between the impacts for boys and for girls was not statistically significant. This 
means the finding must be interpreted with caution. Some impacts were registered for sub- 
groups defined by special education status, but these were inconsistent and did not tell a clear 
story. The researchers sought to examine the impacts for particular subgroups of schools as well 
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as students (for example, those schools where teachers reported receiving more FAST-R 
coaching) but could not do so because the subgroups were too small to yield reliable results. 



Interpreting the Findings 

Since the FAST-R program did not, in general, demonstrate improvements over the sta- 
tus quo, one possible conclusion is that the intervention is no more effective at increasing student 
achievement than the tools that teachers in BPS are already using. But it is also possible that the 
specific circumstances under which it operated in Boston undercut the likelihood that the 
initiative would generate significant impacts. 

In order for an initiative to register positive impacts, one or both of two conditions must 
be in place: The initiative must be implemented in a reasonably strong way, and/or it must 
represent a distinct contrast from the services that are available to individuals who do not 
participate in the initiative. 

FAST-R was implemented as intended in the study schools, but it was not very inten- 
sive. The intervention was designed to be flexible and to provide as much or as little coaching to 
individual schools as administrators and teachers sought. One consequence is that many 
teachers reported getting only a few hours of coaching over the course of the year. Although, as 
noted above, teachers valued the coaching they did receive, the amount may simply have been 
insufficient to affect teaching and learning more than would otherwise have occurred. Further- 
more, FAST-R was implemented in an environment where a great deal was otherwise occur- 
ring. Teachers at comparison schools who responded to the survey indicated that they had 
received at least as much professional development and spent as much time analyzing student 
data as their counterparts at the FAST-R schools. Given this confluence of circumstances, it is 
perhaps not surprising that FAST-R did not have an effect on student achievement above and 
beyond what was happening in similar Boston schools. 

At the same time, it is possible that FAST-R was an efficient way of providing profes- 
sional development to teachers (assuming that such professional development makes a positive 
difference). Those FAST-R teachers who responded to the survey reported consistently that 
they had spent less time on various kinds of professional development than their counterparts at 
the comparison schools (although the differences were not statistically significant), yet their 
students achieved comparable gains. 

However one interprets the impact findings, this single study in a single district — one 
with rich professional development opportunities — should not be taken as the last word on the 
potential of FAST-R. Much more remains to be discovered about how teachers can best learn to 
use student data to improve their instmction and boost students’ achievement. 
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Chapter 1 

Introduction 



The No Child Left Behind Act (NCLB), passed in 2001, places an emphasis on improv- 
ing the learning of all students and mandates annual testing of reading in the third through 
eighth grades. The NCLB reflects a growing national effort to learn about effective instructional 
practices for building reading skills — in all grades but especially for young children — that 
originated from recommendations made by the National Reading Panel.' Many districts and 
schools are using the data generated by annual state assessments that are required for accounta- 
bility purposes to guide district- and school-level changes aimed at boosting achievement in 
reading. Moreover, a growing number of districts and schools have sought more targeted, de- 
tailed, and timely data that teachers can use to improve instruction in their classrooms. Districts 
are developing or purchasing formative assessments aligned with their state standards and ad- 
ministered several times during the school year to obtain useful information about the skills of 
individual students and the effectiveness of instruction.^ 

Formative assessments are tests or activities that measure learning and provide feed- 
back that is then used to adapt teaching practices to meet student needs.^ Learning is in large 
part driven by exchanges between teachers and their students, and formative assessments pro- 
vide data to gauge these exchanges. According to one study, elementary school educators who 
use formative assessments as part of an English language arts (ELA) curriculum cite them as the 
main source of evidence about their students’ reading achievement and progress."' An advantage 
of using formative assessments is the early diagnosis of reading difficulties, which, when identi- 
fied in young students, can help to pinpoint opportunities for remediation in early grades.^ 

Despite their potential for improving students’ achievements, little rigorous research has 
been done on the impacts of formative assessments and instmction informed by the data that 
such assessments generate (or “data-driven instruction”), particularly in reform-rich urban 
school districts. This report seeks to address this knowledge gap regarding the use of formative 
assessments in reading. It evaluates the Eormative Assessments of Student Thinking in Reading 
(EAST-R) initiative, a program implemented in some 50 schools within the Boston Public 
Schools system (BPS) that trains teachers to use formative assessments of reading to inform 
their literacy instmction. The evaluation, funded by the William and Flora Hewlett Foundation, 



'National Institute of Child Health and Human Development (2000). 
^Mumane and Sharkey (2004). 

^Black and Wiham (1998). 

"'Meisels and Piker (2000). 

^Snow, Bums, and Griffin (1998). 
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includes an impact analysis to examine the impact of FAST-R on student achievement in read- 
ing and a process analysis to examine professional development initiatives in FAST-R schools 
and in a group of schools designated as comparison schools for the impact analysis. 

Previous external and internal evaluations of FAST-R have highlighted the promise of 
the intervention. These evaluations were qualitative studies based on data collected from sur- 
veys, site visits, classroom observations, and interviews with school personnel. In an evaluation 
of the FAST-R pilot program, researchers found that the initiative presented a “compelling pic- 
ture of the ways in which authentic formative assessments can lead to teacher knowledge that 
informs instmction.”® In a subsequent evaluation, researchers concluded that FAST-R was able 
to provide timely and valuable formative data in a way that teachers could use and that enabled 
them to better understand student thinking and adjust instruction accordingly.’ 



District Reform Efforts in Boston and the Role of FAST-R in 
Boston Public Schools 

Over the years, BPS has created numerous strategies and initiatives to improve instruc- 
tion and student academic performance in its schools.® In 1996, BPS implemented a reform plan 
called “Focus on Children” that emphasized a standards-based reform approach, including ef- 
forts to achieve high standards, focus on literacy and mathematics instmction, and provide tar- 
geted professional development for principals and teachers. In 2001, BPS implemented a 
second reform plan, “Focus on Children II,” based on its “Whole-School Improvement” reform 
model.® The model is a collaborative process that aims to produce measurable improvement in 
student performance and is organized around six key guiding principles, called “The Six Essen- 
tials” (see Box 1.1), which BPS utilized during the school years covered by this study.'® 

The strong focus on high-quality instmction in BPS motivated the district to undertake 
efforts to enhance opportunities for data-driven instmction and learning from student work. One 
of those efforts is FAST-R, a joint initiative of BPS and the Boston Plan for Excellence (BPE). 
A not-for-profit organization that works to design, pilot, and implement district reform efforts 
by collaborating with the central office and with individual schools, BPE was responsible for 



"Neufeld and Schwartz (2004). 

’Chrismer (2005). 

^Descriptions of BPS district reform efforts are based on information gathered from the BPS Web site 
(www.bostonpublicschools.org). 

^Boston Public Schools (2008a). 

'^Boston Public Schools (2008a). As of fall 2008, BPS uses “The Seven Essentials.” 
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Box 1.1 

Boston Public Schools 

“The Six Essentials for Whole-School Improvement” 



• ONE: Effective Instruction — Focus on literacy and mathematics. 

• TWO: Student Work and Data — Use student work and data to identify student needs, 
improve instruction, and assess progress. 

• THREE: Professional Development — Focus on professional development to offer teachers 
and principals the skills they need to improve instmction. 

• FOUR: Shared Leadership — Identify and replicate best practices for instmction. 

• FIVE: Resources — Align aU resources with the instmctional focus. 

• SIX: Families and Communities — Engage families, communities, and partners to 
support Whole-School Improvement. 



creating, piloting, and disseminating FAST-R in BPS schools. The FAST-R program trains 
teachers to use instmctional strategies based on formative assessment data that build students’ 
skills in reading and understanding texts. The program is intended to complement the district’s 
existing literacy professional development that teachers reported receiving in the 2006-2007 
school year. Teachers in FAST-R schools are primarily given additional targeted training in 
learning how to analyze student assessment data to tailor their reading instmction. Box 1.2 out- 
lines the process for using the FAST-R program in BPS schools. 

FAST-R originated during the 2003-2004 school year as the formative assessments pilot 
and was used on a trial basis in 21 schools." Beginning in the 2005-2006 school year, FAST-R 
underwent full-scale program implementation. All BPS schools were invited to submit proposals 
to participate in FAST-R. Schools were chosen by BPE based on their expressed need to learn 
how to better use data to guide reading instmction. In addition, central office administrators gave 
input about which schools were most suitable for the program. According to BPE, as of the 2007- 
2008 school year, 50 BPS elementary, middle, and high schools use FAST-R regularly. 



"Neufeld and Schwartz (2004). 
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Box 1.2 

The FAST-R Process in Boston Public Schools 

• Teachers administer a FAST-R assessment designed by the Boston Plan for Excellence 
(BPE) to students. 

• Teachers submit the EAST-R assessments to BPE for grading. BPE returns one detailed 
class report showing each individual student’s performance. 

• Teachers receive assistance from BPE instructional data coaches to learn how best to 
analyze the class reports. 

• Data coaches help teachers use the assessment results to tailor their reading instmction 
practices in ways that will strengthen students’ areas of weakness. 



The FAST-R Program 

The FAST-R program begins with a series of short, low-stakes student assessments that 
incorporate question items aligned with the Massachusetts Comprehensive Assessment System 
(MCAS),'^ an annual state high-stakes test for students in grades 3 through 8 and grade 10 that 
is used to measure the performance of both students and schools. FAST-R assessments provide 
information on students’ strengths and weaknesses in English language arts by measuring, pri- 
marily, two skills that readers use to construct meaning from texts: finding evidence to deter- 
mine explicit meaning and making inferences to determine implicit meaning. Both skills are 
central to reading comprehension and are frequently assessed on the MCAS. FAST-R formative 
assessments consist of reading passages from various literary genres, writing styles, and authors. 
While some content is based on former MCAS reading passages and questions, other question 
sets and all teacher support materials were created especially for FAST-R. Other types of for- 
mative assessments are used throughout BPS, but the FAST-R formative assessments are avail- 
able only to schools participating in the FAST-R program. 

Teachers are encouraged to use FAST-R as a tool to identify and target student needs. 
Toward this end, teachers at a given school administer the FAST-R assessments to their classes 
several times over the course of the school year, according to a timetable that each school de- 
termines. BPE notes that, depending on the teachers’ purposes for giving the FAST-R assess- 
ments, some schools administer them every 3 to 6 weeks, while others administer them every 
10 to 12 weeks over the course of the 10-month school year. FAST-R is designed to be flexible, 
so that teachers can choose which assessments to administer depending on their lesson plans. 
BPE staff grade the assessments for teachers and produce analytic reports that are designed to 



'^Descriptions of the FAST-R program are based on information gathered from the BPE Web site 
(www.bpe.org) and staff members. 
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be clear and easy to use. The reports contain detailed data for each student that are intended to 
help teachers notice and respond to patterns in the performance of individual students and stu- 
dent subgroups, and to let teachers see how students performed on the various items. The re- 
ports also explain what incorrect answers signify about each student’s level of understanding, 
and teachers can easily identify how far off students were from selecting the correct answer. 

FAST-R instructional data coaches are an integral part of the FAST-R program. The data 
coaches are experienced teachers, employed by BPE, who previously used FAST-R in their own 
classrooms. Over the course of the study, BPE typically had one elementary school coach, one 
middle school coach, and one high school coach on staff. Data coaches meet periodically with the 
principal and teachers during the school year. They meet with teachers in a group or individually 
to review the data, to discuss how the findings can be used to modify instruction, and to evaluate 
students’ progress since the last EAST-R assessment administration. The aim is to build a data- 
driven culture in which every adult in the school shares a sense of responsibility for each student’s 
progress in learning and to link assessment and instmction in a continuous cycle. 

The EAST-R program is centered on two of the six BPS “essentials” listed in Box 1.1: 
(1) examination of student work and data, and (2) professional development. The focus on stu- 
dent work and data encourages teachers to analyze various kinds of data to inform their instruc- 
tional practices and work with individual students. In addition, as already described, EAST-R 
provides professional development through sessions with an instructional data coach, although 
this is only one of numerous professional development opportunities available to teachers in 
BPS. Within a EAST-R school, the program may be used with specific grade levels depending 
on the needs of the student body and decisions by school leadership and teachers. The EAST-R 
program also assists schools and teachers in the use of formative assessments, but many schools 
not implementing the EAST-R program are also promoting the use of student data to inform 
instruction. In addition, professional development activities referenced in this report represent 
district and school-based training sessions, regularly scheduled teacher meetings to review stu- 
dent work, instructional strategies workshops, work with a literacy coach, work with an instmc- 
tional data coach, and any related activities pursued by all teachers in the district. 



The Evaluation and the Central Research Questions 

This report presents the findings of an impact analysis and a process analysis to shed 
light on the EAST-R program in BPS during the study period. 

Impact Analysis 

The study uses a comparative interrupted time series design to compare the impacts of 
EAST-R on 21 schools that implemented the program in their third- and/or fourth-grade classes 
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during the 2005-2006 and 2006-2007 school years with 36 control schools that did not imple- 
ment FAST-R during these years.'^ This is the first evaluation of FAST-R that compares stu- 
dents’ reading performance over time in schools that implemented the program with the reading 
performance of demographically similar students in schools where the program is not present. 

The impact analysis addresses the following research questions: 

• What is the effect of FAST-R on overall achievement in reading, as meas- 
ured by the MCAS and the Stanford Achievement Test, version 9 (SAT-9), 
reading assessments? 

• What is the impact of FAST-R on students’ ability to draw inferences and to 
provide evidence for their thinking based on a specific reading passage, as 
indicated by items measuring those skills on the MCAS reading assessment? 

• What is the impact of FAST-R on different subgroups of students (for exam- 
ple, by socioeconomic status, gender, prior academic achievement, and so 
forth)? 

Process Analysis 

Aspects of program implementation are examined through surveys that were adminis- 
tered to principals and teachers in the FAST-R and non-FAST-R schools in the spring of 2007. 
This process analysis examines the extent to which FAST-R schools differ from schools that 
have not implemented FAST-R in terms of the extent and types of professional development 
teachers receive and teachers’ responses to this assistance. 

The process analysis will address the following research questions: 

• Where does FAST-R fit into the array of school improvement efforts being 
undertaken within BPS? What school improvement efforts (including other 
efforts to examine data) are in place in both FAST-R and non-FAST-R 
schools? 

• To what extent do teachers in FAST-R and non-FAST-R schools use data to 
inform their thinking about students and to change their teaching strategies? 



*^The comparative interrupted time series design compares the change in one or more outcomes (such as 
reading test scores) over a period of time preceding and following the introduction of an intervention for a 
study group with the change over the same period of time for a similar comparison group that did not receive 
the treatment. 
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The Contents of This Report 

The remainder of this report is organized into four chapters. Chapter 2 describes the 
comparative intermpted time series research design methodology used for the impact analysis. 
The chapter also discusses the sampling methodology used for the surveys that were adminis- 
tered to principals and teachers. 

Chapter 3 discusses the professional development activities in FAST-R and non-FAST- 
R schools highlighted by the findings of the principal and teacher surveys. The chapter also 
considers how teachers perceived the utility of the FAST-R intervention for their instructional 
practices. 

Chapter 4 describes the findings from the impact analysis of FAST-R with regard to 
student achievement, exploring the range of student outcomes on the MCAS and the SAT-9 
reading assessments. In addition, the chapter reports on an analysis to measure the impact of 
FAST-R on students’ ability to make inferences and find evidence while reading. Lastly, sub- 
group analyses to compare the effect of FAST-R on various groups of students (by, for exam- 
ple, gender and socioeconomic status) are discussed. 

Chapter 5 presents the overall conclusions that may be drawn from the study’s analyses 
and their implications for the use of formative assessments and data-driven instruction to im- 
prove reading skills. 
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Chapter 2 

Evaluation Research Design 



MDRC’s evaluation of the Formative Assessments of Student Thinking in Reading 
(FAST-R) program sought to measure the impact of the FAST-R program on student achieve- 
ment in reading and to look at the impact findings within the context of other activities related 
to professional development and data-driven instmction in the Boston Public Schools system 
(BPS). Accordingly, the research design includes two main components. The first section of this 
chapter describes the methods used to analyze the impact of the FAST-R program over and 
above the status quo — that is, beyond what occurred in the comparison schools, where 
FAST-R was not implemented. The second section of this chapter discusses the survey meth- 
odology used to collect data about teachers and principals at both FAST-R and non-FAST-R 
schools regarding their professional development activities and use of data to inform their read- 
ing instruction. Furthermore, the survey findings may suggest how FAST-R fits into the wider 
initiatives of BPS. 



Impact Analysis 

The evaluation focuses on identifying the impact (or ultimate effects) of FAST-R on 
student reading achievement (the “outcome”) over and above what would have occurred in the 
absence of the program. The goal of this study is to determine whether FAST-R is more effec- 
tive, less effective, or about as effective in improving student outcomes relative to the mix of 
reforms that would have occurred without it. 

The Analytic Approach 

The most challenging aspect of any analysis is to determine the nature of the “counter- 
factual” — that is, what would have happened in the absence of the program. The most reliable 
technique to establish the counterfactual is to randomly assign subjects (students or schools) to a 
treatment group that has access to the program or to a control group that does not have such 
access. Random assignment ensures that any differences between the two groups, before the 
treatment occurs, are due to chance. For this reason, the control group outcomes can be used as 
a reliable estimate of the outcomes that would have been observed among treatment group 
members without the program. In this case, however, the FAST-R program was implemented 
before the study was designed and schools were not selected at random for the FAST-R pro- 
gram, making use of an experimental design impossible. 
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Instead, an alternative research design known as the comparative interrupted time series 
is applied.' This approach compares the change in student outcomes over a period of time pre- 
ceding and following the introduction of the intervention for schools that adopted FAST-R with 
the corresponding change over the same period for similar comparison schools that did not 
adopt it. The ideal conditions for this type of analysis include consistent student outcomes for 
multiple pre-intervention baseline years and multiple post-implementation follow-up years as 
well as the inclusion of multiple schools that implemented the FAST-R program and compari- 
son schools that did not implement the program.^ As can be seen from Box 2.1, the large num- 
ber of both schools implementing FAST-R and comparison schools that did not implement the 
program, as well as consistent longitudinal test score data, led to a particularly strong execution 
of this analysis model. 

The approach rests on two comparisons. The first compares the FAST-R schools’ stu- 
dent outcomes before and after the program was implemented, since — absent any school or 
district intervention or change in the school’s student body — the best predictor of educational 
outcomes in a given school is the history of that school’s student outcomes. The second com- 
parison controls for any effects of district-wide interventions or changes in student populations 
occurring simultaneously with the FAST-R program implementation by comparing the change 
in educational outcomes of students at FAST-R schools with the change in outcomes for similar 
schools that did not implement FAST-R. 

Comparison 1 : Deviation from the Trend 

The first comparison measures differences in outcomes before and after program im- 
plementation at the group of schools where the FAST-R program was put in place. First, a pre- 
intervention “baseline” period is measured for each outcome in question. Since it is possible that 
a general growth or decline in the outcome measures could be occurring during the five baseline 
years, it is important to take into account these baseline “trends” when calculating program ef- 
fects. For this study, a trend was used to measure the outcomes for the five baseline years 
(2000-2001 through 2004-2005) and predict those outcomes for the two years (2005-2006 
through 2006-2007) following program implementation. The “deviation from the trend” is the 
difference between the predicted outcome and the actual average outcome for each program 
school during each follow-up year. 



'Bloom (2003); Snipes (2003). 

^The inclusion of multiple pre-intervention years allows researchers to capture the growth or decline in 
student test scores over time, while including more than one post-implementation year allows researchers to 
analyze the effects of the program beyond the first year after implementation. 
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Box 2.1 



Ideal Elements for an Interrupted Time Series Anaiysis Modei 
and Actuai Components in FAST-R Anaiysis 

• Consistent student outcomes for multiple pre-intervention baseline years: 

Five years of baseline outcomes on MCAS (spring 2001-2005) and SAT-9 (fall 2001- 
2005) standardized tests 

• Consistent student outcomes for multiple post-implementation follow-up years: 

Two years of follow-up outcomes on MCAS (spring 2006 and 2007) and SAT-9 (fall 
2006 and 2007) 

• Multiple schools that implemented the FAST-R program: 

Twenty-one FAST-R schools (20 in third-grade analysis and 19 in fourth-grade 
analysis) 

• Multiple comparison schools that did not implement the FAST-R program: 

Thirty-six comparison schools (31 in third-grade analysis and 29 in fourth-grade 
analysis) 

NOTES: MCAS = Massachusetts Comprehensive Assessment System; SAT-9 = Stanford 
Achievement Test, version 9. 



The top panel in Figure 2. 1 presents a hypothetical example of this comparison as ap- 
plied to academic achievement test scores. The diagram plots five years of student achievement 
before the program intervention began and two years of student achievement following the pro- 
gram implementation. The asterisks on the left-hand side of the graph represent average 
achievement scores among students in a given program school in the analysis. The solid line 
running through these points represents the trend in scores across all the baseline years. The 
dashed line on the right-hand side of Figure 2. 1 projects the trend occurring in the five baseline 
years into the two follow-up years. This latter trend line serves as the benchmark against which 
post-implementation outcomes can be measured. The points marked FI and F2 represent hypo- 
thetical outcomes in each follow-up year for the program school. The distance between each of 
these points and the predicted trend represents the deviation from the trend. The dotted lines 
above and below the projected baseline represent the 90 percent confidence interval — that is, 
the range of values within which there is a 90 percent probability that the projected trend actual- 
ly falls. To the extent that the average outcome value in any follow-up year falls outside the 
confidence interval for the predicted trend, the estimated deviation from trend for that year is 
statistically significant. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 



Figure 2.1 

Illustration of a Hypothetical Interrupted Time Series Analysis: 
Difference in the Deviation from the Trend 



Program schools 




Comparison schools 




NOTES: The confidence interval is the range of values within which there is a 90 percent probability that the 
projected trend actually falls. Each notch on the x-axis represents the end of a school year in a hypothetical study. 
The vertical dashed line represents the break between the baseline (pre-intervention) and follow-up (post- 
implementation) periods. Fi and F 2 represent hypothetical outcomes in each follow-up year. 
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Looking at the program schools alone, however, it would be impossible to determine 
how much of the observed change from the projected trend was attributable to FAST-R and 
how much it reflected changes in other circumstances that could have affected all Boston public 
schools during the period after FAST-R’ s implementation. If, during the same period as the 
FAST-R program, other major changes and/or reforms that are unrelated to FAST-R also oc- 
curred, these changes or reforms, rather than the FAST-R program, could be driving the ob- 
served effects. 

Comparison 2: Difference in the Deviation from the Trend 

The second step in the analysis, therefore, is to measure the change in the outcome over 
the same period at a set of Boston public schools that resemble each FAST-R school in terms of 
baseline achievement levels and demographics but that did not implement the initiative. The 
change over time at these comparison schools provides an estimate of how student performance 
would have changed at the FAST-R schools in the absence of the FAST-R reform but including 
other reforms that the district may have instituted. The bottom panel of Figure 2. 1 illustrates the 
deviation from the trend at a hypothetical set of comparison schools. This deviation from the trend 
for the comparison schools is measured in the same manner as was done for the program schools. 

The impact is the difference between the deviation from the trend for program schools 
and the deviation from the trend for the comparison schools.^ To the extent that the deviations 
from the trend at program and comparison schools systematically differ in a positive direction, 
one can conclude that FAST-R had an effect on student achievement, over and above what 
would have occurred in the absence of the program. To the extent that the deviations do not dif- 
fer, such a conclusion is not supported. 

In short, the approach is to: 

• Calculate the deviation from the trend at a set of schools implementing 
FAST-R. 

• Calculate the deviation from the trend at a set of carefully chosen comparison 
schools. 

• Compare the deviations from the trend at FAST-R schools with the devia- 
tions from the trend at comparison schools. 



^The analytic model used to estimate impacts was structured to calculate each school’s deviation from its 
respective trend, which was constructed using the five years of basehne data. It then compares the program 
schools’ deviations with the deviations of the comparison schools to estimate impacts for the overall program. 
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Selection of FAST-R Schools and Comparison Schools 

The analysis described above is conducted separately for third- and fourth-grade stu- 
dents/ The 21 FAST-R elementary and K-8 (kindergarten through eighth grade) schools in- 
cluded in the analyses (20 schools in the third-grade analysis and 19 schools in the fourth-grade 
analysis) were fully and consistently implementing the FAST-R program in third- and/or fourth- 
grade classrooms with the support of the Boston Plan for Excellence (BPE) during the 2005- 
2006 school year/ Schools that served special populations exclusively or did not test third- and 
fourth-grade students during any of the baseline or follow-up years were not included. Almost 
all of the schools included in the program group were also actively implementing EAST-R in 
the 2006-2007 school year.® 

The comparison schools are included in the analysis in order to provide an estimate of 
the progress that would have occurred at the program schools without the EAST-R program. 
Therefore, the goal in choosing comparison schools was to find a set of BPS schools that, in the 
absence of the intervention, would be expected to perform similarly to the schools that imple- 
mented the EAST-R program. A set of BPS elementary schools that had not been exposed to the 
EAST-R program during the 2000-2001 through 2006-2007 school years was chosen as a poss- 
ible comparison group. These schools were then matched to specific EAST-R schools using a 
set of criteria. The most accurate predictor of future performance on an outcome is usually pre- 
vious performance on that same outcome. Therefore, prior academic achievement was the pri- 
mary criterion by which comparison schools were selected. Still, it could be argued that, even 
with similar prior achievement patterns, schools that serve different student populations may 
evolve differently over time, particularly in response to local events or district policies. There- 
fore, comparison schools were also selected because they serve students who are demographi- 
cally similar to those in the EAST-R schools in terms of race/ethnicity, eligibility for free or re- 
duced-price lunch, and the percentage of special education students and students with limited 
English proficiency (EEP). 



"^Third and fourth grades were chosen for the study because of the availabihty of MCAS data, but also be- 
cause reading for comprehension begins to be especially important at these grade levels, as reading materials 
become more complex. 

®These 21 schools include 10 schools that were involved in the FAST-R pilot program conducted during 
the 2003-2004 and 2004-2005 school years. Since most of the schools involved in the pilot program did not 
fully and consistently implement the FAST-R program during these years and since the pilot version of the 
program did not include many of the elements existing in the later FAST-R program, the pilot years were 
treated as baseline for all schools in the study. A sensitivity test of the pilot program was conducted and found 
that there were no statistically significant positive impacts at the schools involved in the pilot during the years 
of the pilot program. 

®Two schools in the third-grade analysis and one school in the fourth-grade analysis were not actively im- 
plementing FAST-R in the 2006-2007 school year. 
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These matching criteria were applied to the third and fourth grades separately using the 
data from the 2000-2001 through 2002-2003 school years.’ Since no two schools were exactly 
alike on all these criteria, each FAST-R school was matched with as many comparison schools 
as were suitable. At the same time, each comparison school was allowed to match to more than 
one FAST-R school where applicable.® Most FAST-R schools matched closely to other BPS 
schools on both demographic (percentage minority, poverty, special education, and LEP) and 
test score variables (Stanford Achievement Test, version 9 [SAT-9], and Massachusetts Com- 
prehensive Assessment System [MCAS]).^ In some cases, FAST-R schools did not match to 
comparison schools using the strictest matching criteria, but, upon widening those parameters, 
every school matched at least one comparison, or non-FAST-R, school.'° 

Characteristics of FAST-R and Non-FAST-R Schoois 

For the comparative intermpted time series model to be effective, the comparison 
schools must provide a reliable indicator of how the student outcomes are likely to respond to 
district- wide policies or events during the years FAST-R is being implemented. To make this 
assumption, it is important that the FAST-R and non-FAST-R schools’ student populations be 
similar. Table 2.1 presents characteristics of the third- and fourth-grade students in the FAST-R 
schools and their comparison schools in the five years prior to the full-scale implementation of 



’The 2003-2004 and 2004-2005 school years were not used for matching comparison schools because 
some of the FAST-R schools had been exposed to the FAST-R pilot program during those years. 

®The 36 comparison schools (31 schools in the third-grade analysis and 29 schools in the fourth-grade 
analysis) were weighted so that the data from all the non-FAST-R schools matching a particular FAST-R 
school were combined to create a “pseudo-school” that most closely matched that FAST-R school. To take into 
account that a comparison school could also be matched to several FAST-R schools, all the weights given to a 
specific non-FAST-R school for matching with several different corresponding FAST-R schools were 
summed. 

^More specifically, comparison schools were selected using the strictest matching criteria: the third- and 
fourth-grade students’ average SAT-9 and MCAS scores were no more than one-fourth of a standard deviation 
above or below the FAST-R school’s average scores; the percentage of white students, percentage of students 
receiving free or reduced-price lunch, and percentage of limited English proficient or formerly limited English 
proficient students in third and fourth grade were no more than 10 percentage points above or below the 
EAST-R school’s percentage; and the percentage of third- and fourth-graders in special education was no more 
than 15 percentage points above or below the EAST-R school’s percentage. 

*®For the third grade, four EAST-R schools did not match to any comparison schools using the above pa- 
rameters. To match these schools to at least one comparison school, the difference allowed for free and re- 
duced-price lunch was increased from 10 to 15 percentage points, and the difference in schools’ average 
MCAS and SAT-9 scores was increased from one-fourth to one-half of a standard deviation above or below 
the EAST-R school’s average score. Eor fourth grade, five EAST-R schools did not match on the above para- 
meters but matched to at least one comparison school when the difference in schools’ average MCAS and 
SAT-9 test scores was increased from one-fourth to one-half of a standard deviation above or below the 
EAST-R school’s average score. A sensitivity test was conducted where these schools were excluded from the 
analyses. The patterns of impacts were similar to those found in the analyses with these schools included. 
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Table 2.1 



Characteristics of Students at FAST-R and Non-FAST-R Schools: 
Pre-Intervention Years, SY 2000-2001 to 2004-2005 





FAST-R 


Non-FAST-R 






Characteristic 


Schools 


Schools'* 


Difference 


P- Value 


Third-erade students 










Age (years) 


8.7 


CO 

od 


0.0 


0.742 


Race/ethnicity (%) 










Black 


46.3 


58.3 


-12.0 


0.043 ** 


White 


14.7 


11.9 


2.8 


0.356 


Hispanic 


31.6 


25.5 


6.2 


0.258 


Asian 


6.8 


3.5 


3.3 


0.137 


Other 


0.2 


0.3 


-0.1 


0.466 


Male (%) 


54.6 


51.6 


3.0 


0.004 *** 


Eligible for free lunch (%) 


76.4 


79.1 


-2.7 


0.268 


Limited English proficiency (%) 


13.5 


11.1 


2.4 


0.414 


Special education (%) 


22.5 


20.1 


2.4 


0.327 


Student reading achievement” 










MCAS ELA (distance from proficiency) 


-4.5 


-5.3 


0.9 


0.194 


MCAS ELA proficient or above (% of students) 


36.8 


32.6 


4.2 


0.210 


SAT-9 Total Reading (scaled score) 


578.9 


577.1 


1.8 


0.557 


Total number of schools 


20 


31 






Fourth-erade students 










Age (years) 


9.8 


9.8 


0.0 


0.335 


Race/ethnicity (%) 










Black 


46.4 


55.8 


-9.4 


0.119 


White 


11.0 


11.3 


-0.3 


0.906 


Hispanic 


35.4 


27.5 


7.9 


0.208 


Asian 


6.4 


4.6 


1.8 


0.437 


Other 


0.3 


0.4 


0.0 


0.711 


Male (%) 


52.3 


52.2 


0.1 


0.934 


Eligible for free lunch (%) 


80.8 


80.7 


0.0 


0.985 


Limited English proficiency (%) 


11.0 


8.6 


2.3 


0.314 


Special education (%) 


21.4 


20.9 


0.4 


0.871 


Student reading achievement” 










MCAS ELA (scaled score) 


230.3 


229.2 


1.0 


0.389 


MCAS ELA proficient or advanced (% of students) 


26.4 


24.0 


2.4 


0.437 


SAT-9 Total Reading (scaled score) 


604.9 


603.9 


1.0 


0.714 


Total number of schools 


19 


29 






(continued) 
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Table 2.1 (continued) 

SOURCE: MDRC calculations from individual student school records from the Boston Publie Schools' district 
office. 

NOTES: A two-tailed t-test was applied to the difference in baseline means between EAST-R and non-EAST-R 
schools. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 pereent. 

Third-grade MCAS ELA = Massachusetts Comprehensive Assessment System, English Language Arts, 
Reading Comprehension, Grade 3; fourth-grade MCAS ELA = Massachusetts Comprehensive Assessment 
System, English Language Arts, Grade 4; SAT-9 = Stanford Achievement Test, version 9; SY = school year. 

“EAST-R schools were matched with multiple comparison schools. All comparison schools matched to a 
particular EAST-R school are weighted so that the total weight of all the matching non-FAST-R schools equals 
one. Since each comparison school could also be matched to several FAST-R schools, all the weights of a 
particular non-FAST-R school are summed. 

'’The SAT-9 was administered in the fall of students' third-grade year, and the MCAS was administered in the 
spring of that year. 

“The SAT-9 was administered in the fall of students' fourth-grade year, and the MCAS was administered in 
the spring of that year. 



FAST-R in Boston." Approximately 50 percent of the students, across both grades and in both 
FAST-R and non-FAST-R schools, were black, and 30 percent were Hispanic. Almost 80 per- 
cent of the students qualified for free or reduced-price lunches, and approximately 20 percent 
were identified as having special education needs. As can be seen from the table, the samples of 
FAST-R schools and non-FAST-R schools used in these analyses are similar. In third grade, 
there are statistically significant differences between groups in percent black and percent of stu- 
dents who were male. In fourth grade, there are no statistically significant differences between 
the FAST-R schools and the comparison schools. 

Controlling for Changes in School Composition 

This analysis takes into account the fact that FAST-R and non-FAST-R schools may ex- 
perience a change in the composition of their student populations over the seven years included in 
the analysis. For example, neighborhoods may undergo demographic changes or the mles govern- 
ing school assignment at the district may change. It is also possible that the implementation of 
FAST-R could affect the population of students directly or indirectly. In order to account for any 
systematic changes in the characteristics of the students at particular schools over time, the analy- 
sis incorporates individual student characteristics within a given school into the analytic model. 
Specifically, the analytic model includes controls for students’ prior achievement, race/ethnicity, 
student age, gender, special education status, LEP and former LEP status, and whether or not a 
student is receiving free or reduced-price lunch used as a proxy for poverty. 



"The non-FAST-R schools are weighted in the same manner as in the impact analysis (see note 8). 
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Selection Bias 

Because the selection of schools for FAST-R was not random, schools that imple- 
mented FAST-R could differ in major (but unmeasured) ways from those that did not mount the 
intervention. Schools applied to participate in and then were selected for the FAST-R program 
by BPE staff based on a set of criteria, as described below. Later, comparison schools were cho- 
sen from a pool of schools that either applied for the program and were not accepted or never 
applied. The findings could be biased if the comparison schools do not represent what would 
have happened to the FAST-R schools in the absence of the program. The choice of a school to 
apply or not apply for the program may cause bias. For example, a school that was later ac- 
cepted into the FAST-R program may have applied because the administration felt the school 
was struggling in this area and needed assistance, while a school that did not apply to be in the 
FAST-R program, and later became a non-FAST-R school, may have felt the school’s current 
programs in this area were successful. This difference in the FAST-R and non-FAST-R schools 
may cause the estimated impacts to be smaller than the tme impacts. Still, it is also possible that 
schools that applied for the program were more motivated to change their professional devel- 
opment activities and strengthen their reading programs than schools that did not apply, which 
might cause the estimated impacts to be larger than the tme impacts. 

Bias also could have occurred during BPE’s selection of schools. If schools were cho- 
sen because of their capacity for successful implementation and progress, it is possible that the 
program effects could be overestimated. On the other hand, if schools were chosen because they 
had the most need for support and a low capacity for success, it is possible that the program ef- 
fects could be underestimated. In the selection process, BPE took into account several factors, 
including both the quality of the applications (perhaps an indication of a school’s higher capaci- 
ty for success) and the need for assistance (perhaps an indication of a school’s lower capacity 
for success). This mixed selection process helped to reduce systematic bias, since schools were 
chosen for the intervention using two somewhat opposing factors. BPE staff also noted that 
some schools that wrote persuasive applications did not always implement the initiative consis- 
tently, suggesting that high capacity at one point in the process did not ensure that a school’s 
staff would be consistently and adequately able to carry out the reform. Although selection bias 
is an issue that cannot be fully resolved, the variety of motivations schools may have had in 
choosing to apply for the program and the variety of criteria that BPE used in the selection 
process make it unlikely that these effects would overwhelm the findings. 

Data Sources and Outcome Measures 

The impact analysis relies on individual student records obtained from BPS. This data- 
base includes information on individual student performance on standardized tests as well as 
student data regarding race/ethnicity, school enrollment status, eligibility for free or reduced- 
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price lunch, special education status, and LEP status. The goal of this analysis is to demonstrate 
whether or not FAST-R has an effect on student reading achievement, which is measured using 
three outcomes (described below) from two standardized tests — the MCAS and the SAT-9. 
Two literacy skills integral to the FAST-R program, making inferences from and finding evi- 
dence in reading passages, are also measured separately. 

MCAS English Language Arts Total Score 

The Massachusetts Comprehensive Assessment System is the high-stakes state test in 
Massachusetts that is administered to all students in the third through eighth grades and tenth 
grade in the spring of each year. Student scoring on this test is the key impact outcome of inter- 
est because it is the state test that is used to monitor school accountability and student achieve- 
ment. 



The MCAS Enghsh language arts (EFA) test for third-graders — titled, “Enghsh Fan- 
guage Arts, Reading Comprehension, Grade 3” — is based on learning standards in the two 
content strands of the Massachusetts English Language Arts Curriculum Framework (June 
2001):'^ “Fanguage” and “Reading and Fiterature.”'^ The third-grade test is not measured in 
scaled scores (that is, adjusted for comparative purposes) because it does not include material 
that can reliably distinguish between proficient and advanced work. The Massachusetts De- 
partment of Education does publish performance-level thresholds for third-grade test scores, 
including a threshold indicating proficiency. The threshold scores are measured to maintain the 
same performance standards across all the years included in this study. To get the most accu- 
rate measure of each student’s score that is comparable to other students’ scores across all years 
included in the analysis, each student’s raw (that is, unadjusted) score was measured as the dis- 
tance from that year’s proficiency threshold.'^ 

The MCAS English language arts test for fourth-graders — titled, “English Fanguage 
Arts, Grade 4,” and comprising two sections. Composition and Reading Comprehension — is 



'^Published by the Massachusetts Department of Education. Web site; 
www.doe.mass.edu/frameworks/current.html. 

*^The name of the MCAS third-grade Enghsh language arts test prior to 2007 was “Reading, Grade 3.” 

'"'Massachusetts Department of Education (2007). 

'^There are two question formats on the MCAS reading test in both third and fourth grade, multiple-choice 
questions and open-ended questions. There are two open-ended questions on the third-grade tests and four on the 
fourth-grade tests. For third-grade students, open-ended questions were not included in the reported raw scores 
until the spring 2006 test, thus changing the make-up of the raw score during the follow-up years. The reported 
raw scores and their corresponding threshold scores for performance levels were used for all years. Although the 
threshold scores were created to maintain performance levels, this change in the make-up of the test scores may 
affect the comparability across pre- and post-implementation years. A sensitivity test was conducted where each 
student’s score was standardized by dividing by the standard deviation for all students in that year. When the anal- 
ysis was run on these standardized outcome variables, similar patterns of impacts were found. 
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based on learning standards in the three content strands of the Massachusetts English Language 
Arts Curriculum Framework (2001):'® “Language,” “Reading and Literature,” and “Composi- 
tion.”'’ The scaled score was used as the outcome variable for fourth grade. 

MCAS Percentage At or Above “Proficient” Level in Reading 

Massachusetts defines a set of four standards for the MCAS based on students’ perfor- 
mance on the reading test. The performance standards are defined as follows: (1) advanced (for 
fourth grade) and above proficient (for third grade), (2) proficient, (3) needs improvement, and 
(4) warning. These are the most policy-relevant thresholds with which to compare student read- 
ing performance because schools are judged based on these measures. The MCAS percentage at 
or above “proficient” is used to investigate changes around this policy-relevant threshold. Im- 
pact estimates are presented as the percentage of students in the school performing at or above 
the proficient level. 

Making Inferences and Finding Evidence 

To further understand students’ reading skills, the FAST-R initiative focuses on two 
key literacy skills: making inferences and finding evidence in text. According to Boston Plan 
for Excellence (BPE) staff, some 60-75 percent of MCAS items tap into these two skills. To 
better understand whether or not the EAST-R program is affecting either of those individual 
literacy skills, impact estimates were calculated on the set of specific MCAS items, as identified 
by BPE, that measure each skill. To best measure these subsets of items over several years, 
where the difficulty of the questions in each subset changes from year to year, the difference 
between each student’s score and the state average was calculated for each item in a subset.'* 



'^Published by the Massachusetts Department of Education. Web site: 
www.doe.mass.edu/frameworks/current.html. 

'^The name of the Reading Comprehension section of the Grade 4 MCAS ELA prior to 2007 was “Lan- 
guage and Literature.” 

'*The reading test consists of multiple-choice and open-ended questions. Each multiple-choice question is 
worth one raw score point and is counted as one if the student answered it correctly and zero if the student 
answered incorrectly. If the student answered it correctly, the percentage of students who answered the ques- 
tion correctly across the state is subtracted from one. Otherwise, the state percentage of students who answered 
the question correctly is subtracted from zero. Eor example, if 47 percent of students across the state answered 
a particular multiple-choice question correctly, a student answering that same question correctly would receive 
a scaled score of 0.53 (or 1 - 0.47); a student who did not answer that question correctly would receive a scaled 
score of -0.47 (or 0 - 0.47). Each open-ended question is worth four points, and students can receive a score 
between zero and four. Eor these questions, the average score statewide is subtracted from the student’s score. 
Eor example, if a student had a raw score of 3 on an open-ended question and the average raw score statewide 
was 2.7, the student’s scaled score would be 0.3. 
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Then the scores for all the items in each subset were averaged for each student to create a meas- 
ure of each subset’s average distance from the overall state mean.'® 

SAT-9 Total Reading Score 

By design, the formative assessments used in FAST-R resemble MCAS items. Many 
FAST-R questions are either drawn directly or adapted from the MCAS. The FAST-R ques- 
tions are also multiple-choice in form, as are many of the MCAS questions. The similarity be- 
tween the two assessments, however, raises an interpretive problem. If impacts are found on 
students’ MCAS scores, it would not be clear whether those impacts are connected to students’ 
familiarity with the test or to changes in teachers’ instmctional practices resulting from the use 
of interim assessments. To address this problem, the Stanford Achievement Test, version 9 
(SAT-9) was used to assess students’ reading skills. 

The SAT-9 is a norm-referenced test, or a test that compares each student’s score with a 
national sample, that BPS uses as a diagnostic tool in the fall of each school year. Generally 
considered to be a broad test, the SAT-9 Total Reading test covers a wide range of reading- 
related content, including phonemic awareness, decoding, phonics, vocabulary, and comprehen- 
sion. Students’ scaled scores on the SAT-9 were used as a secondary measure of student reading 
achievement because the assessment is not similar in form to the FAST-R assessments. Still, 
there are some issues with the validity of this test as an outcome measure since it is given in the 
fall of each year and is not a high-stakes test (students have no particular incentive to do well 
and teachers have no particular incentive to encourage them to try hard). For the third-grade 
analyses, the fourth-grade fall SAT-9 Total Reading score was used as an outcome, and for the 
fourth-grade analyses, the fifth-grade fall SAT-9 Total Reading score was used. 

Analyses of the FAST-R Program by Gender, Student Socioeconomic 
Status, Baseline Reading Achievement, and Special Education Status 

To better understand who is most affected by the FAST-R program, a secondary set of 
analyses was done on the effects of FAST-R on student achievement by student socioeconomic 
status, gender, and students’ pre-program reading achievement. The effects on student achieve- 
ment for those students receiving free or reduced-price lunch were calculated to establish whether 
the FAST-R program was more or less successful in helping students of lower socioeconomic 
status. Effects on female and male students were analyzed separately to assess whether FAST-R 
was affecting the reading achievement of boys and girls differently. Fourth-grade students were 



'®A sensitivity test was conducted in which each student’s score was standardized by dividing it by the 
standard deviation for all students in that year. When the analysis was run on these standardized outcome 
variables, similar patterns of impacts were found. 
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grouped by their pre-program reading achievement to explore whether FAST-R was more ef- 
fective for lower- or higher-achieving students. Students’ SAT-9 Total Reading scores from the 
fall of their third-grade year were used. Students were split into two groups: students who per- 
formed at or above the “Level 3, Proficient” performance standard and those who performed 
below this proficiency threshold.^' Finally, analyses were done separately on students receiving 
special education services and those not receiving special education services in both third and 
fourth grades.^^ 



Process Analysis 

Methodology 

The process analysis is used to provide contextual information for the impact analysis 
findings in two ways. First, it clarifies the extent of reading instmction and professional devel- 
opment efforts in the non-FAST-R schools. Since the non-FAST-R schools are used to estimate 
what the FAST-R school student achievement would have been without the implementation of 
the FAST-R program, a better understanding of instmctional improvement initiatives at non- 
FAST-R schools can aid in explaining the magnitude of any impacts. Second, the process anal- 
ysis compares principal and teacher attitudes in FAST-R and non-FAST-R schools with regard 
to school functioning and resources. 

Surveys were used to collect data for the process analysis. These surveys were adminis- 
tered to teachers and principals working in BPS schools that participate and do not participate in 
the FAST-R program. 

Survey of Teachers 

The survey of teachers contained both close-ended multiple-choice questions and open- 
ended questions. It asked teachers in both FAST-R and non-FAST-R schools about their parti- 

1995, proficiency standards were established for the SAT-9. The four performance standards include 
Level 4, Advanced; Level 3, Proficient; Level 2, Basic; and Level 1, Below Basic. See Harcourt Brace & 
Company (1997). 

^'Baseline reading achievement subgroups were not created for third-grade students because of missing 
data for two schools where, during one year of the study, no students took the SAT-9 test in the fall. 

^^Low sample sizes in parricular schools precluded analysis of student achievement by students’ 
race/ethnicity. Missing data made it impracrical to measure student outcomes by LEP status. School-level sub- 
group analyses were also attempted, including comparing student-level outcomes in schools where teachers 
reported receiving more or less FAST-R coaching and where BPE judged that the implementation of FAST-R 
had been stronger or weaker. However, these school-level subgroup analyses were not possible because there 
were too few schools where teachers reported receiving five or more hours of coaching during the school year 
or where BPE judged the school to have fully implemented the program. 
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cipation in professional development activities that helped them to understand their students’ 
thinking skills, such as district and school-based training sessions, teacher meetings, activities 
with a literacy coach, and activities pursued by teachers on their own (such as graduate school 
courses). 

Survey items for teachers in FAST-R schools also included questions about specific 
components of the FAST-R program that were not available to teachers in the non-FAST-R 
schools, such as the extent of their work with an instructional data coach. Teachers’ answers to 
the multiple-choice questions were computed to compare teachers in FAST-R and non-FAST-R 
schools on the amount of professional development they received, their attitudes toward the use- 
fulness of that professional development, and their use of student data to inform their instmc- 
tional practices. The open-ended questions were used as a source for teachers’ descriptions of 
their experiences with professional development activities, using data to inform their teaching 
practices, and using the FAST-R assessments with their students. 

Administration of paper surveys to third- and fourth-grade teachers, including special 
education and Sheltered English Instruction teachers,^^ was handled by Survey Research Man- 
agement (SRM), a survey firm. The surveys were administered in spring 2007. SRM attempted 
to survey teachers in all FAST-R schools and all potential comparison non-FAST-R schools. 
Teachers were given a $15 Borders gift card as an honorarium for volunteering to complete the 
surveys. The final survey sample consisted of 169 teacher surveys — 84 FAST-R teachers 
representing 14 of the 21 FAST-R schools in the impact analysis and 85 non-FAST-R teachers 
representing 18 of the 34 non-FAST-R schools in the impact analysis. (FAST-R teachers are 
teachers who are working in participating schools. Non-FAST-R teachers work in schools that 
did not participate in FAST-R.)The response rates for the teacher surveys were 53 percent for 
FAST-R schools and 56 percent for non-FAST-R schools.^”* These low response rates can pos- 
sibly be explained by events in the district. At the time of the survey administration, the Boston 
Teachers Union was negotiating a new contract, and teachers may have been disinclined to un- 
dertake any additional activities beyond what was required of them. Given the low response 
rate, the teachers who took part in the survey cannot be considered representative of all teachers 
in schools included in the impact analysis sample. The teacher survey findings, therefore, can be 
viewed as suggestive only, rather than definitive, in showing the similarities and differences 
between FAST-R and non-FAST-R teachers. 

Table 2.2 displays the characteristics of the FAST-R and non-FAST-R teachers who 
are included in the survey sample. On average, FAST-R teachers had 10.7 years of experience 



^^Sheltered English Instruction teachers work with LEP students. 

^‘'Due to the low response rate for the teacher survey, researchers analyzed responses from the third- and 
fourth-grade teachers together in the EAST-R and non-EAST-R schools. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 2.2 



Characteristics of FAST-R and Non-FAST-R Teachers 
Sampled in the Teacher Survey 



Characteristic 


FAST-R Non-FAST-R 
Teachers Teachers 


Difference 


P- Value 


Average number of years in teaching 


At current school 


6.6 


8.8 


-2.2 


0.036 ** 


In Boston Public Schools 


9.4 


13.4 


-4.0 


0.006 *** 


Reading/English language arts 


10.7 


13.9 


-3.2 


0.030 ** 


Grade taught in the 2006-2007 school year (%) 


Third grade 


48.2 


47.6 


0.6 


0.482 


Eourth grade 


44.4 


38.1 


6.3 




Third and fourth grades 


7.4 


14.3 


-6.9 




Average number of students taught English language arts 


18.5 


17.9 


0.7 


0.675 


Type of classes taught (%) 


Regular education 


51.8 


51.8 


0.0 


0.585 


Inclusion 


10.8 


1.2 


9.6 




Resource room special education 


6.0 


5.9 


0.1 




Substantially separate special education 


10.8 


21.2 


-10.4 




Sheltered English Immersion (SEI) 


12.1 


11.8 


0.3 




Other 


8.4 


8.2 


0.2 




Total number of teachers 


84 


85 







SOURCES: MDRC calculations are from teacher surveys administered to third- and fourth-grade teachers 
in elementary Boston Public Schools during April 2007. 

NOTES: Rounding may cause slight discrepancies in calculating sums and differences. The sample size 
reported represents the number of teachers who filled out a survey. Teachers for whom values are missing are 
not included in the calculations. 

A two-tailed t-test was used to measure statistical significance. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 



teaching Reading/English Language Arts compared with 13.9 years of experience for non- 
FAST-R teachers. Both FAST-R and non-FAST-R teachers, on average, taught English Lan- 
guage Arts to the same number of students in the 2006-2007 school year. On average, FAST-R 
teachers had 6.6 years of experience teaching at their current school compared with 8.8 years of 
experience for non-FAST-R teachers. Both of these differences are statistically significant. De- 
spite the variations, both groups of schools were composed of experienced teachers. 
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Survey of Principals 

The survey of principals consisted of questions inquiring about principals’ leadership 
and their teachers’ participation in various professional development activities, such as district- 
and school-based training sessions, teacher meetings, activities with a literacy coach, and activi- 
ties pursued by teachers on their own. Principals in FAST-R schools were also asked about the 
benefits of the FAST-R program for their teachers and students. 

MDRC staff members contacted the principals and conducted the surveys over the tele- 
phone beginning in spring 2007.^^ The survey was e-mailed to principals prior to their phone 
interview to allow them to follow along. They were given a $15 Borders gift card as an hon- 
orarium for volunteering to complete the surveys. 

The final survey sample consisted of 24 surveys of principals, representing 24 out of 57 
total schools in the impact analysis. Because the survey was conducted near the end of the 
school year, many principals could not be reached, which contributed to low response rates. The 
response rates for the principal surveys were 57 percent for FAST-R schools and 33 percent for 
non-FAST-R schools.^'’ On average, FAST-R principals who answered the survey had 6.4 years 
of experience as principal at their current school, compared with 4.4 years of experience for 
non-FAST-R principals, but the difference is not statistically significant. Given the low re- 
sponse rates, principals who took part in the survey cannot be considered representative of all 
principals in schools within the impact analysis sample. 

The general results of both surveys are discussed in Chapter 3. As a result of the low re- 
sponse rates, any conclusions drawn from the survey responses are suggestive only and not 
conclusive. 



^^Researchers believed it would be more efficient to administer the survey over the phone to principals, 
given the small number of principals in the targeted sample. 

^'’Due to low responses rates, survey responses from FAST-R and non-FAST-R principals were aggre- 
gated for analysis. 
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Chapter 3 

FAST-R and Other Professional Development Activities 

in Boston Public Schools 



The Boston Public Schools system (BPS) is a refonn-rich urban district that has under- 
taken multiple initiatives to improve instmction and student performance. The districtwide fo- 
cus on professional development provides school leaders and teachers with a variety of training 
opportunities and related activities offered by local educational institutions, professional organi- 
zations such as the International Reading Association, and other BPS partners, including the 
Boston Plan for Excellence (BPE).' BPE is the district’s primary partner in improving instmc- 
tion and in changing central academic policies.^ BPS’s objective for professional development 
is to align teacher professional development with school and district goals and the specific 
learning needs of the student body.^ As part of its effort to meet that objective, BPE — in part- 
nership with BPS — created FAST-R with the goal of providing timely data to teachers to help 
students meet proficiency standards in reading. 

FAST-R provides one kind of professional development training to teachers by offering 
them an opportunity to work with an instmctional data coach, who helps them analyze forma- 
tive assessment data, and to build on their existing work with school-based literacy coaches. 
Coaching is one among a plethora of professional development resources that teachers can util- 
ize within the district, although only the FAST-R teachers work with the BPE instmctional data 
coach. 



This chapter presents the analysis of teacher and principal surveys comparing FAST-R 
and non-FAST-R schools’ professional development opportunities and use of student data. In 
addition, it includes a discussion of the implementation of the FAST-R program. Although 
teacher and principal surveys serve as the basis for this analysis, the respondents cannot be con- 
sidered representative of all teachers and principals in schools within the impact analysis sample 
because of the low response rate, as observed in Chapter 2. The survey findings, therefore, 
should be considered suggestive rather than definitive. Still, the analysis in this chapter suggests 
several key findings: 

• FAST-R teachers who took the survey reported, in general, that their 
FAST-R training — that is, professional development activities, including 

'Boston Public Schools (2008b). 

^According to the BPS Web site (www.bostonpublicschools.org), BPE has a significant role in instruction 
reform. 

^Boston Public Schools (2004). 
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coaching in data interpretation and reading instruction — was useful and 
contributed to their understanding of student assessment data. 

• There are few differences between FAST-R and non-FAST-R teachers with 
respect to the amount of professional development they received during the 
2006-2007 school year. Overall, both groups reported that professional de- 
velopment was helpful to some extent in informing their reading instmction 
practices. 

• Survey data from the principals suggest that most principals in both FAST-R 
and non-FAST-R schools support and encourage teachers to engage in pro- 
fessional development activities. 

• Non-FAST-R teachers who took the survey spent more time analyzing the 
2006 Massachusetts Comprehensive Assessment System (MCAS) test re- 
sults than did FAST-R teachers who took the survey. Otherwise, there is not 
much variation between the FAST-R and non-FAST-R teachers with respect 
to the time they spent on formative assessment data analysis and perceived 
usefulness of analyzing both kinds of data. 



FAST-R Teacher Training 

FAST-R teachers participated in targeted professional development with an insfruction- 
al data coach, who taught them how to analyze formative assessment data and how to utilize 
data to inform their insfructional practices. Generally, FAST-R teachers reported that their 
FAST-R training was useful and enhanced their understanding of the student assessment data. 

Teachers in FAST-R schools reported spending various amounts of time reviewing 
FAST-R data with a FAST-R instmctional data coach. Throughout the school year, one full- 
time data coach worked with elementary school teachers on how to interpret assessment data, 
draw conclusions from the data, and then use those conclusions to inform their work with stu- 
dents. Figure 3.1 displays the number of hours FAST-R teachers reported spending with their 
FAST-R data coach during the 2006-2007 school year. Sixty-one percent of FAST-R teachers 
spent 1 to 5 hours, while only 13 percent of FAST-R teachers spent 1 1 or more hours working 
with a FAST-R data coach. FAST-R teachers reported spending modest amounts of time utiliz- 
ing this key component of the program’s professional development activities. One reason for 
this limited time may be that only one instmctional data coach, assisted sometimes by another 
BPE staff member, is assigned to work with all the FAST-R elementary schools, which limits 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Figure 3.1 

Hours of FAST-R Training with Instructional Data Coach Reported by 
Teachers in FAST-R Schools, SY 2006-2007 
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SOURCES: MDRC calculations are based on responses to surveys administered to third- and fourth-grade 
teachers in elementary Boston Public Schools during April 2007. 

NOTES: The sample size for FAST-R teachers was 84. Rounding may cause slight discrepancies in 
calculating sums and differences. The sample size reported represents the number of teachers who filled out 
a survey. Teachers for whom values are missing are not included in the calculations. 

SY= school year. 



29 




the amount of time that the coach could spend at an individual school. Since teachers spend lim- 
ited time with the data coach, they may utilize other school-based resources, such as the lit- 
eracy coach, who observes and gives feedback to teachers about class lessons that are co- 
designed with the data coach. (The literacy coach is also available to non-FAST-R teachers.) 
Even though FAST-R teachers spent limited amounts of time working with data coaches, they 
reported that having a data coach was a valuable resource. For example, one FAST-R teacher 
stated: “The FAST-R coach was useful in deciphering the data and giving mini-lesson ideas for 
[student] areas of weakness.” 

FAST-R teachers described using FAST-R in several ways to identify students’ English 
language arts skill levels. Table 3.1 presents FAST-R teachers’ perceptions about the FAST-R 
program and how it contributed to their teaching. Eighty-eight percent of FAST-R teachers re- 
ported using FAST-R to assess students’ reading comprehension. The majority of FAST-R 
teachers also reported that FAST-R contributed to their understanding of assessments (60.8 per- 
cent of teachers) and their use of student assessment data to “reflect on” their instmctional prac- 
tices (71.6 percent of teachers). In addition, teachers commented on various ways in which 
FAST-R data helped them explore their students’ ways of thinking about reading. For example, 
one FAST-R teacher stated: 

One of my students did a better job at making inferences [than the] finding evi- 
dence-related questions. This was surprising. I gave him another FAST-R pas- 
sage a week later (same genre), only this time he responded 100 percent cor- 
rectly to finding evidence questions and 60 percent to making inferences, 

[which was] confusing! Yet, it was an opportunity for me to really delve into 
what and how he was thinking about the text. 

FAST-R teachers reported that FAST-R helped them better understand students’ 
thinking and the answers students chose during reading exercises. Figure 3.2 presents 
FAST-R teachers’ responses regarding FAST-R’s effect on their perceptions of students’ abil- 
ities. Eighty-four percent of FAST-R teachers found that FAST-R data confirmed what they 
knew about their students. One FAST-R teacher explained: 

Using FAST-R with my class has been instmmental in providing my students 
with the strategies [and] skills to become better readers and thinkers. While dis- 
cussing the results, students use accountable talk in explaining their reasoning.'* 

This [accountable talk] is powerful because it clarifies misunderstanding and 
empowers them to be active learners. 



'*In class lessons characterized by “accountable talk,” students are required to support and justify their an- 
swers with reasoning and evidence. 



30 




Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 3.1 

FAST-R Teachers’ Uses and Perceptions of the FAST-R Program: 
Boston Public Schools, SY 2006-2007 



FAST-R 

Use of FAST-R Program Teachers 

Teachers reporting using FAST-R for a particular purpose (%) 

Assessment of students' reading comprehension 88.3 

Assessment of students' use of particular reading skill or strategy 87.0 

Way of giving students additional experience with the MCAS 

"testing genre" 83.1 

Part of a Collaborative Coaching and Learning (CCL) cycle 47.3 



Teachers reporting how FAST-R contributed to a "moderate" 



or "great" extent for a particular purpose (%) 

Understanding of how data can be relevant to students' work 74.3 

Understanding of different kinds of assessments and their uses 60.8 

Using data to reflect on practice 71.6 

Sample size 84 



SOURCES: MDRC calculations are from teacher surveys administered to third- and fourth-grade 
teachers in elementary Boston Public Schools during April 2007. 

NOTES: Rounding may cause slight discrepancies in calculating sums and differences. The sample size 
reported represents the number of teachers who filled out a survey. Teachers for whom values are 
missing are not included in the calculations. 

MCAS = Massachusetts Comprehensive Assessment System; SY = school year. 



Professional Development in English Language Arts 

Compared with the amount of professional development that teachers in the FAST-R 
schools reported they had received, teachers in non-FAST-R schools reported receiving as 
much or more professional development. Professional development activities were available to 
both groups of teachers through widespread offerings provided by the Boston Public Schools 
district. Overall, both groups also reported that professional development was helpful to some 
extent in informing their reading instmction practices. 

Table 3.2 compares how FAST-R and non-FAST-R teachers perceive the time spent 
on and usefulness of various types of professional development activities during the 2006- 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Figure 3.2 

Teachers’ Reports of How FAST-R Data Affected Their Perceptions of Students 





Students' FAST-R data helped me question subgroup patterns in 
my classroom. 




Students' FAST-R data challenged my views of some of 
them. 




Students' FAST-R data helped me understand their 
thinking as they do their daily reading. 




Students' FAST-R data helped me understand their 
thinking and their answers. 



Students' FAST-R data confirmed what I knew about 
them. 



SOURCES: MDRC calculations are based on responses to surveys administered to third- and fourth-grade 
teachers in elementary Boston Public Schools during April 2007. 

NOTES: The sample size for FAST-R teachers was 84. Rounding may cause slight discrepancies in 
calculating sums and differences. The sample size reported represents the number of teachers who filled out a 
survey. Teachers for whom values are missing are not included in the calculations. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 3.2 
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I to 5 hours 17.5 17.6 -0.1 

6 to 10 hours 11.3 6.8 4.5 

I I or more hours 37.5 51.4 -13.9 




FAST-R FAST-R 

le of Professional Development Teachers Teachers Difference 
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2007 school year. Generally, similar proportions of FAST-R and non-FAST-R teachers reported 
that the various types of professional development in which they had participated over the 
course of the year were “somewhat useful” or “very useful.” FAST-R teachers and non-FAST- 
R teachers similarly reported that the professional development they had received contributed to 
their understanding of data to both guide instmction and build on students’ strengths. 

The majority of teachers in both groups found professional development provided by 
BPS to be useful and were especially enthusiastic about Collaborative Coaching and Learning 
(CCL), the district’s school-based professional development delivery model. CCL is used in all 
BPS schools to encourage and create opportunities for collaboration among teachers. Teachers 
work in teams during eight-week cycles to learn about instructional strategies for reading and 
writing. CCL coaches share these strategies with teachers, who then take turns demonstrating 
them to their colleagues. 

Table 3.3 displays the percentage of BPS principals who reported that they encouraged 
their teachers to participate in professional development activities and to analyze student data 
during the 2005-2006 and 2006-2007 school years. Generally, BPS principals encouraged their 
third- and fourth-grade teachers “to a moderate extent” or “to a great extenf ’ to participate in 
these activities. BPS principals in both FAST-R and non-FAST-R schools supported profes- 
sional development activities as part of the instmctional reforms taking place throughout the 
district. 



Table 3.4 compares FAST-R and non-FAST-R teachers’ responses regarding the spe- 
cific ways in which professional development helped their instmctional practices. Both FAST-R 
and non-FAST-R teachers reported that professional development strengthened their instmc- 
tional practices by helping them conduct classroom discussions and create assignments focused 
on reading skills like finding evidence in and making inferences from text. The FAST-R inter- 
vention stresses these skills, so it is not surprising that FAST-R teachers receive this guidance. 
Non-FAST-R teachers may receive similar guidance focused on these skills from other profes- 
sional development resources such as their school literacy coaches. Literacy coaches are a 
common resource among FAST-R and non-FAST-R teachers because they are members of the 
school staff in BPS. 



Examining the Use of Data at FAST-R and Non-FAST-R Schoois 

The MCAS test and formative assessments provide the two main forms of student per- 
formance data available to BPS teachers. The MCAS, which is an annual high-stakes state test 
used for accountability, measures students’ achievement on state standards. Formative assess- 
ments are low-stakes assessments administered frequently during the year. Through the FAST-R 
program, FAST-R teachers receive formative assessments designed by BPE to distribute to 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 3.3 

Proportion of Principals Who Encouraged 
Professional Development Activities and Data Analysis: 

Boston Public Schools, SY 2005-2006 and SY 2006-2007 

BPS 

Type of Professional Development and Data Analysis Activity Principals 



Principals reporting that they encouraged professional development 
to a "moderate" or "great" extent i%i 

Curriculum-specific professional development 69.6 

Observing other teachers' classrooms 78.3 

Collaborative Coaching and Learning (CCL) or work with a literacy coach 78.3 

Work with an instructional data coach 65 .2 

Teacher's own professional development 69.6 

Other professional development 79.0 

Principals reporting that they encouraged data analysis 
to a "moderate" or "great" extent i%l 

2006 MCAS ELA results 100.0 

Formative assessments 95.2 

Sample size 24 



SOURCES: MDRC calculations are Ifom surveys administered to principals in FAST-R and non-FAST-R 
schools in elementary Boston Public Schools during April 2007. 

NOTES: Rounding may cause slight discrepancies in calculating sums and differences. The sample size reported 
represents the number of principals who filled out a survey. Principals for whom values are missing are not 
included in the calculations. 

MCAS ELA = Massachusetts Comprehensive Assessment System, English Language Arts; SY= school year. 



students. Non-FAST-R teachers may have access to formative assessments through supporting 
materials that accompany their reading curriculum guides. 

Table 3.5 shows how FAST-R and non-FAST-R teachers compare on the time spent 
and perceived usefulness of analyzing data. Non-FAST-R teachers reported spending more time 
than FAST-R teachers analyzing the 2006 MCAS results. This difference is statistically signifi- 
cant. On the other hand, there is not a statistically significant difference in the time non-FAST-R 
and FAST-R teachers reported analyzing formative assessments. Although not statistically sig- 
nificant, non-FAST-R teachers were more likely than FAST-R teachers (87 percent compared 
with 81 percent) to report that analyzing the MCAS results was ‘‘somewhat” or “very” 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 3,4 



Ways in Which Professional Development Helped FAST-R and Non-FAST-R Teachers: 

Boston Public Schools, SY 2006-2007 



Type of Professional Development 


FAST-R 

Teachers 


Non- 

FAST-R 

Teachers 


Difference 


Teachers renortine orofessional develooment heloed 
to a "moderate" or "ereat" extent t%) 








Conduct classroom discussions focused on reading skills 
like finding evidence and making inferences 


76.2 


75.0 


1.2 


Create assignments focused on reading skills 
like finding evidence and making inferences 


73.8 


75.9 


-2.1 


Group students strategically for guided reading 
or instruction in particular reading strategies 


65.8 


66.7 


-0.9 


Sample size 


84 


85 





SOURCES: MDRC calculations are from teacher surveys administered to third- and fourth-grade teachers in 
elementary Boston Public Schools during April 2007. 

NOTES: Rounding may cause slight discrepancies in calculating sums and differences. The sample size 
reported represents the number of teachers who filled out a survey. Teachers for whom values are missing are 
not included in the calculations. 

A chi-square test was used to measure statistical significance. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * =10 percent. The chi-square test may not be valid due to small sample sizes 
within the cross-tabulation distribution. 



useful. FAST-R teachers were slightly more likely than non-FAST-R teachers (95 percent com- 
pared with 92 percent) to report that analyzing formative assessments was “somewhat” or 
“very” useful for teaching, although, again, the difference was not statistically significant. One 
FAST-R teacher elaborated, stating. 

Analyzing student data has informed me about the areas in [the] students’ 
knowledge that need to be reinforced or focused on. It doesn’t just inform me 
on what/how to teach to get students ready for assessments but also [what is 
needed] for them to be academically competent and well rounded. 

It is possible that FAST-R teachers spend less time analyzing data because the student 
performance reports generated from the FAST-R assessment are designed to be user-friendly. 
These FAST-R reports were created to provide student data in a more efficient manner than 
other standard reading curriculum assessments. They contain color-coded graphics to allow 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 3,5 



FAST-R and Non-FAST-R Teachers’ Reported Usefulness of and 
Time Spent Analyzing Data: Boston Public Schools, SY 2006-2007 







Non- 






FAST-R 


FAST-R 




Type of Data 


Teachers 


Teachers 


Difference 


Number of hours teachers reported analvzina data t%) 








2006 MCAS ELA results 








None 


10.1 


5.0 




1 to 5 hours 


73.4 


53.8 


19.6 


6 to 10 hours 


8.9 


23.8 


-14.9 


1 1 or more hours 


7.6 


17.5 


-9.9 


Formative assessments 








None 


2.5 


3.7 


-1.2 


1 to 5 hours 


48.2 


46.9 


1.3 


6 to 10 hours 


18.5 


18.5 


0.0 


1 1 or more hours 


30.9 


30.9 


0.0 


Teachers renortina analvzine data as "somewhat" 
or "verv" useful t%) 








2006 MCAS ELA results 


80.5 


87.2 


-6.7 


Formative assessments 


95.0 


92.3 


2.7 


Sample size 


84 


85 





SOURCES: MDRC calculations are from teacher surveys administered to third- and fourth- grade teachers in 
elementary Boston Public Schools during April 2007. 

NOTES: Rounding may cause slight discrepancies in calculating sums and differences. The sample size reported 
represents the number of teachers who filled out a survey. Teachers for whom values are missing are not included 
in the calculations. 

A chi-square test was used to measure statistical significance. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * =10 percent. The chi-square test may not be valid due to small sample sizes 
within the cross-tabulation distribution. 

MCAS ELA = Massachusetts Comprehensive Assessment System, English Language Arts; SY= school year. 



teachers to identify quickly and easily which questions students answered incorrectly — indivi- 
dually, within subgroups, and overall as a class. 

Table 3.6 displays the various school personnel with whom teachers were likely to re- 
view student data. Overall, both FAST-R and non-FAST-R teachers reported they were most 
likely to review data with other teachers, followed by their principals and then their literacy 
coaches. None of these differences is statistically significant. Both FAST-R and non-FAST-R 



38 








Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 3,6 



FAST-R and Non-FAST-R Teachers, by School Personnel 
with Whom They Reviewed Data: Boston Public Schools, SY 2006-2007 



School Personnel with Whom Teachers Reviewed Data (%) 


FAST-R 

Teachers 


Non- 

FAST-R 

Teachers 


Difference 


Princinal 


Yes 


61.9 


60.7 


1.2 


No 


38.1 


39.3 


-1.2 


Assistant prindnal 


Yes 


14.3 


15.5 


-1.2 


No 


85.7 


84.5 


1.2 


Literacv coach 


Yes 


57.1 


59.5 


-2.4 


No 


42.9 


40.5 


2.4 


Other teachers 


Yes 


88.1 


86.9 


1.2 


No 


11.9 


13.1 


-1.2 


Sample size 


84 


85 





SOURCES: MDRC calculations are from teacher surveys administered to third- and fourth-grade 
teachers in elementary Boston Public Schools during April 2007. 

NOTES: Rounding may cause slight discrepancies in calculating sums and differences. The sample 
size reported represents the number of teachers who filled out a survey. Teachers for whom values 
are missing are not included in the calculations. 

A chi-square test was used to measure statistical significance. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * =10 percent. The chi-square test may not be valid due 
to small sample sizes within the cross-tabulation distribution. 

SY = school year. 



teachers stated in their open-ended survey responses that many opportunities exist for reviewing 
data with other teachers, including CCL meetings and grade-level meetings. 

A variety of initiatives emphasizing the use of data to improve instmction are diffused 
throughout BPS. Overall, surveyed teachers represented in both FAST-R and non-FAST-R 
schools have reported taking advantage of district-wide professional development opportunities. 
The FAST-R program is one of many supplemental tools for instmctional support available to 
teachers working in the BPS district. 
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Chapter 4 



The Effects of FAST-R on Reading Achievement 
for Boston Pubiic Schoois’ 

Third- and Fourth-Grade Students 



The main goal of the FAST-R impact analysis is to assess whether FAST-R improved 
achievement in reading for students in the Boston Public Schools system (BPS) over and above 
what would have been observed in the absence of the program. The analysis compares changes 
in student achievement at a set of FAST-R schools with changes in achievement at a set of 
comparison schools that did not implement the FAST-R program. As discussed in the first 
chapter, BPS has incorporated a wide array of educational and teacher professional develop- 
ment programs and has made strengthening the use of assessment data a district-wide goal. 
Therefore, FAST-R schools are being compared with schools that have programs that may in- 
clude some similar components, making it harder for the FAST-R program to attain positive 
impacts beyond the status quo. 

This chapter presents the impacts of FAST-R on reading achievement for third- and 
fourth-grade students during the 2005-2006 and 2006-2007 school years. The analysis in this 
chapter suggests several key findings: 

• Although the estimated impacts of FAST-R on third- and fourth-grade stu- 
dent reading achievement as measured by the Massachusetts Comprehensive 
Assessment System (MCAS) are generally positive, these findings are not 
statistically significant. Therefore, the effect of FAST-R on student reading 
achievement in both third and fourth grade over and above what occurred in 
similar schools throughout the district is indeterminate. 

• As a secondary measure of student reading achievement, the effect of FAST- 
R on the Stanford Achievement Test, version 9 (SAT-9) given in the fall fol- 
lowing a full year of the FAST-R intervention was also measured. The esti- 
mated impacts on student reading achievement on the SAT-9 were mixed and 
were not statistically significant, suggesting that FAST-R did not improve 
these outcomes beyond what would have happened without the program. 

• The FAST-R formative assessment focuses on two types of questions that 
make up a large percentage of the questions on the MCAS English Language 
Arts (ELA) test, questions that ask students both to make inferences from and 
to find evidence in a two-page reading passage (see Appendix D). The 
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FAST-R program did not have a statistically significant impact on students’ 
ability to answer either type of question. 

• Subgroup analyses were conducted on boys and girls, students of differing 
socioeconomic status, students performing above or below the “proficienf ’ 
level on a baseline reading test, and students receiving and not receiving spe- 
cial education services. There are very few statically significant impacts 
across all of these analyses. Since these few impacts do not create a coherent 
story and the number of impacts is less than might be expected as a result of 
chance, they should be interpreted cautiously. 



Effects of the FAST-R Program on Third-Grade Students’ 

Reading Achievement 

The effects of FAST-R on third-graders’ reading achievement are measured primarily 
by looking at students’ performance on the MCAS. In third grade, the MCAS ELA total score is 
measured as the deviation from the proficiency threshold. This is because MCAS scores for 
third-graders are not measured in scaled scores, which are conversions of students’ raw scores 
into scores that are comparable across different versions of the test. The raw scores, or actual 
number of items a student answered correctly on the test, that are provided by the Massachusetts 
Department of Education are not comparable over the five years before the intervention and the 
two years after implementation that are included in the study. The threshold between scores in 
the “needs improvement’’ and “proficienf’ categories, published by the Massachusetts Depart- 
ment of Education each year, offers a method for comparing students’ test scores across the 
years of the study. For instance, the threshold score between “needs improvemenf ’ and “profi- 
cient’’ for the third-grade reading test in spring 2006 was 37 raw score points. If a student ans- 
wered 40 of the questions correctly, her distance from the proficiency threshold would be +3. If 
a student scored 35 on the test, his distance from the proficiency threshold would be -2. 

Figure 4.1 (page 44) illustrates the average change in third-grade students’ MCAS EEA 
scores at a “representative’’ FAST-R and non-FAST-R school over the five pre-implementation 
years and then projects those trends into the post-implementation years.' As can be seen from 

'To create Figure 4.1, the average baseline scores and the trends over time for “representative” FAST-R and 
non-FAST-R schools were calculated. (“Representative” reflects what is being portrayed based on the steps de- 
scribed here.) To do this, the student-level baseline characteristics were first centered by subtracting the school- 
level mean from each characteristic for each student. For example, if the student is male, the value for whether or 
not a student is male equals one (it equals zero if the student is female). If 52 percent of the students in the school 
this student attends were male, the student’s centered value for male would equal 0.48 (1 - 0.52). The model was 
ran using every student’s centered baseline characteristics to find the average baseline score and trend for the av- 
erage student in each school. The average baseline score and the trend for a representative FAST-R school are the 

(continued) 
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this figure, the trends in student test scores for both the representative FAST-R and non-FAST- 
R schools during the five years before implementation were nearly flat (that is, 0.0 for the repre- 
sentative FAST-R school and 0.03 for the representative non-FAST-R school), suggesting that 
third-grade student achievement on the MCAS, during the five years before FAST-R was im- 
plemented, was steady in both FAST-R and non-FAST-R schools. 

Impact findings of the effects of FAST-R on third-grade students’ reading achievement 
for both the 2005-2006 and 2006-2007 school years are presented in Table 4.1 (page 46). In the 
first year after implementation, the average distance from proficiency for the students in the rep- 
resentative FAST-R school was -7.8. This means that students in FAST-R schools scored, on 
average, 7.8 points below the proficiency threshold in the spring of 2006. The deviation from 
the trend for FAST-R students (-2.1) in 2006 is the amount that the distance from proficiency 
(-7.8) for students in FAST-R schools deviates from the expected outcome (-5.7). It suggests 
that in the spring of 2006, third-grade students in FAST-R schools scored 2.1 points below what 
would have been expected given the trend in test scores for the five years before the FAST-R 
program was implemented. Students in the non-FAST-R schools scored an average of 8.1 
points below the proficiency threshold in the spring of 2006. The deviation from the trend for 
non-FAST-R students was -2.2, suggesting that students in non-FAST-R schools scored 2.2 
points below the expected outcome. Therefore, in both FAST-R and non-FAST-R schools, av- 
erage reading achievement on the MCAS for third-grade students was lower than expected in 
the first year after FAST-R was implemented, given the trends in test scores during the five 
years before the program began for each group. 

Since the decline in test scores from pre- to post-implementation is similar for students 
in both the FAST-R and non-FAST-R schools (that is, the difference is not statistically signifi- 
cant), the decline in student achievement cannot be attributed to the FAST-R program.^ The es- 
timated impact of FAST-R is the difference between the deviation from the trend for students in 
FAST-R and non-FAST-R schools. Since the main difference between these two groups is that 
one participated in the FAST-R program and the other did not, the estimated impact can be con- 



weighted averages of the results for all FAST-R sehools. The average baseline seore and the trend for a represent- 
ative non-FAST-R sehool are the weighted averages of the results for all non-FAST-R sehools. 

^As suggested in Chapter 2, the seoring for the third-grade MCAS ELA raw seore was different for the pre- 
implementation and post-implementation years, hr all seven years, students answered two open-ended questions 
but these questions were not ineluded in the ealculations of the students’ raw seores until 2006. Although this 
ehange in the test make-up is aceounted for in the MCAS performanee standards used to ereate the “distanee from 
profieieney” measure, it may have affeeted the deviations from the trends for both follow-up years, eausing some 
of the drop in test seores. But, sinee the impaets are measured as the eomparison between FAST-R and non- 
FAST-R students and the scoring methods for these two groups are the same, the change does not affect the im- 
pact analysis. As a sensitivity test, the scores were standardized by dividing each student’s score by the standard 
deviation of all student scores during that year. When the analysis was run on these standardized scores, the pat- 
terns in outcomes were similar. 
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MCAS distance from proficiency MCAS distance from proficiency 



Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Figure 4.1 

Trends in Third-Grade Students’ MCAS ELA Scores 
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Figure 4.1 (continued) 

SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000- 
2001 and 2006-2007 academic years. 

NOTES: To calculate the trends and intercepts for a representative FAST-R and non-FAST-R school, the 
student-level baseline characteristics were first centered by subtracting the school-level mean from the 
characteristic for each student. The model was run using these centered variables to find the trend and intercept 
for the average student in each school. The trend and intercept for each school was then multiplied by the 
percentage of the total students in the school. The results for all FAST-R schools were summed to find the trend 
and intercept for a "representative" FAST-R school, and the results for all non-FAST-R schools were summed to 
find the average trend and intercept for a "representative" non-FAST-R school. 

The deviation from trend for each post-implementation year for the non-FAST-R schools is regression- 
adjusted. The deviation from trend for each post-implementation year for the FAST-R schools is the regression- 
adjusted impact added to the non-FAST-R schools' deviation from trend. The average student achievement 
scores are calculated by subtracting the deviation from trend from the predicted average for a given follow-up 
year, calculated using the average school trend and intercept. 

MCAS ELA = Massachusetts Comprehensive Assessment System, English Language Arts, Reading 
Comprehension, Grade 3. 



sidered the effect of the FAST-R program. As shown in Table 4.1, the estimated impact of 
FAST-R is 0.1 raw score point during the first year after implementation, which corresponds to 
an effect size of 0.01 standard deviation (or 1 percent of the standard deviation, a measure of the 
variation in the distribution that is used to compare outcomes across variables, of both FAST-R 
and non-FAST-R students’ MCAS scores during the five years before FAST-R was imple- 
mented).^ The difference between the students in the FAST-R and non-FAST-R schools is neg- 
ligible and is statistically indistinguishable, as indicated by the lack of asterisks. Thus, it cannot 
be concluded that the actual impact is different from zero. In the second year after implementa- 
tion, there is a notably larger estimated impact of FAST-R on the third-grade MCAS ELA score 
of 0.9 raw score point, with an effect size of 0. 1 1 standard deviation. Still, this finding is not 
statistically significant and, again, it cannot be concluded that this impact is actually different 
from zero. 

Effect sizes are used to make outcomes across different studies and variables compara- 
ble. Although no absolute standard exists to define whether a specific effect size is large or 
small, there are some guidelines. One way to think about the findings in this report is to com- 
pare them with a typical year of growth for a similar population of students. An analysis of the 
students outcomes on seven standardized reading tests given to students across the country 
found that the average growth in student test scores from second to third grade was an effect 
size of 0.60 standard deviation. This means that students’ general overall growth in reading 
from the spring of second grade to the spring of third grade was approximately 60 percent of a 



^The effect size is a measurement used to compare outcomes across variables and studies. It was calculated 
by dividing the impact by the standard deviation of all students’ scores during the five baseline years. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 4.1 
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(continued) 




SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000-2001 and 2006-2007 
academic years. 
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standard deviation. From third to fourth grade, the effect size was 0.36 standard deviation."* Ef- 
fect sizes of the impacts of the FAST-R program can be compared with these measures of over- 
all growth during these grades. For instance, the effect size on the MCAS English Eanguage 
Arts test for third-grade students in the first year after implementation of 0.01 standard deviation 
is quite small considering that student reading achievement, on average, increases by an effect 
size of 0.60 standard deviation during the third-grade year. Of course, it should be remembered 
that the above impact estimates, and most of the impact estimates found in this study, are not 
statistically significant, and so it is not possible to know whether the tme impacts are actually 
different from zero. 

Another way to think about the outcomes for this study is to compare them with the 
outcomes for other similar types of interventions in similar studies. A meta-analysis (that is, a 
statistical analysis of data combined from multiple studies) of 76 meta-analyses of educational 
interventions found that the mean effect size for elementary school students in these studies was 
0.23 standard deviation for lower grades (first through third) and 0.22 standard deviation for 
higher grades (fourth through fifth). Another meta-analysis of 61 random assignment studies 
(the most rigorous impact design available) of educational interventions found that the mean 
effect size for all the elementary school findings in these studies was 0.33 standard deviation. 
This meta-analysis also found that the mean effect size for random assignment studies con- 
ducted using a broad standardized test (similar to MCAS) was 0.07.^ Again, an effect size, like 
the one seen above, of 0.01 standard deviation is small considering that the average effect size 
for studies using similar measures is at least seven times as large. 

The second measure in Table 4.1 is the percentage of third-grade students who scored at 
a level of proficient or above on the Grade 3 MCAS EEA test. A positive impact on this meas- 
ure would suggest that the program was particularly successful in helping lower-achieving stu- 
dents score at or above their grade level. As can be seen from the table, 27.4 percent of third- 
grade students at a representative FAST-R school scored at the proficient level or above in the 
spring of 2006 — 5.1 percentage points lower than expected, given the trend in the percentage 
of students scoring at that level in the five years before implementation.*’ In a representative 
non-FAST-R school, 27.6 percent of third-grade students scored at or above proficient — 3.5 
percentage points lower than anticipated by the trend in test scores from the five years before 
FAST-R implementation. Although the FAST-R schools show a greater negative deviation 



"*Hill, Bloom, Black, and Lipsey (2007). 

Bloom, Black, and Lipsey (2007). 

^The average yearly baseline trends for students in the representative FAST-R school (0.52 percentage point 
per year) and those in the representahve non-FAST-R school (0.50 percentage point per year) are positive, sug- 
gesting that the percentage of students scoring at or above proficient during the five years before program imple- 
mentation was growing slightly in both FAST-R and non-FAST-R schools. 
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from the trend, the difference in the deviations is not statistically significant, and thus it cannot 
be concluded that the actual impact is different from zero. During the second year after imple- 
mentation, FAST-R schools show a smaller negative deviation from the trend than non- 
FAST-R schools (-4.5 percentage points compared with -5.6 percentage points, respectively). 
Still, the difference in this deviation, or the estimated impact, of 1 . 1 percentage points is not sta- 
tistically significant. 

The final measure in Table 4.1 is the effect of the FAST-R program on the SAT-9 Total 
Reading scaled score given in the fall of the students’ fourth-grade year. The deviation from the 
trend is positive for both FAST-R and non-FAST-R students, suggesting that both groups had 
higher average test scores than expected from the trend in SAT-9 scores during the five years 
before the implementation of FAST-R.^ The deviation from the trend for students in FAST-R 
schools is smaller than the deviation from the trend for students in non-FAST-R schools, mak- 
ing the impact -2.8 scaled score points, with an effect size of -0.07 standard deviation. This 
negative impact is not statistically significant. In the second year after implementation, the esti- 
mated impact on the SAT-9 Total Reading test is -1.1 scaled score points, and is also not statis- 
tically significant. Overall, there are no statistically significant impacts for the FAST-R program 
on the Grade 3 MCAS ELA or SAT-9 Total Reading scores. 



Effects of the FAST-R Program on Fourth-Grade Students’ 
Reading Achievement 

Table 4.2 presents the impacts for both the 2005-2006 and the 2006-2007 school year on 
Grade 4 MCAS ELA and SAT-9 Total Reading scores. In fourth grade, the MCAS ELA test is 
measured in scaled scores. This metric is comparable over the seven years of the study and is used 
in the analysis. The findings for the Grade 4 MCAS ELA scaled score tell a similar story as those 
for third grade. Figure 4.2 (page 52) illustrates the average change in fourth-grade MCAS ELA 
scores for students in a representative FAST-R and non-FAST-R school over the five pre- 
implementation years and then projects those trends into the post-implementation years.* For 



^The average yearly trends for the SAT-9 sealed seores for both the representahve FAST-R sehools (0.20 
sealed seore point per year) and the representative non-FAST-R sehools (0.29 sealed seore point per year) during 
the five baseline years are positive, suggesting that all sehools were showing some progress on this test in the 
years before the FAST-R program was implemented. 

*To ealeulate the trends and intereepts for a representative FAST-R and non-FAST-R sehool, the student- 
level baseline eharaeteristies were first eentered by subtraeting the sehool-level mean from eaeh eharaeteristie for 
eaeh student. The model was run using these eentered variables to find the trend and intereept for the average stu- 
dent in eaeh sehool. The trend and intereept for eaeh sehool were multiplied by the pereentage of the total students 
in the sehool. The results for all FAST-R sehools were summed to find the trend and intereept for a representative 
FAST-R sehool, and the results for all non-FAST-R sehools were summed to find the average trend and intereept 
for a representative non-FAST-R sehool. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 4.2 
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(continued) 




SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000-2001 and 2006-2007 
academic years. 
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with SAT-9 scores reported in the fall of their fifth-grade year. 




MCAS scaled score MCAS scaled score 



Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Figure 4.2 

Trends in Fourth-Grade Students’ MCAS ELA Scores 
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Figure 4.2 (continued) 

SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000- 
2001 and 2006-2007 academic years. 

NOTES: To calculate the trends and intercepts for a representative FAST-R and non-FAST-R school, the 
student-level baseline characteristics were first centered by subtracting the school-level mean from the 
characteristic for each student. The model was run using these centered variables to find the trend and intercept 
for the average student in each school. The trend and intercept for each school was then multiplied by the 
percentage of the total students in the school. The results for all FAST-R schools were summed to find the trend 
and intercept for a "representative" FAST-R school, and the results for all non-FAST-R schools were summed to 
find the average trend and intercept for a "representative" non-FAST-R school. 

The deviation from trend for each post-implementation year for the FAST-R schools is the regression- 
adjusted impact added to the non-FAST-R schools' deviation from trend. The average student achievement 
scores are calculated by subtracting the deviation from trend from the predicted average for a given follow-up 
year, calculated using the average school trend and intercept. 

MCAS ELA = Massachusetts Comprehensive Assessment System, English Language Arts, Grade 4. 



students in both FAST-R and non-FAST-R schools, the trends are positive, suggesting a general 
growth in reading achievement for all students in the study. As with the third grade, the devia- 
tions from the trend for both FAST-R and non-FAST-R students are negative during both the 
2005-2006 and the 2006-2007 school year. This finding suggests that students’ scores were 
generally lower than would be expected given the trend in scores over the five years prior to 
implementation. Still, in both years of the intervention, the negative deviation from the trend is 
smaller for the students in FAST-R schools than for the students in non-FAST-R schools. The 
estimated impacts of FAST-R, as shown in Table 4.2, are positive in both years, but these im- 
pacts are not statistically significant. The estimated impact in the first year of implementation is 
0.8 scaled score point, with an effect size of 0.06 standard deviation. The estimated impact for 
the second year after implementation is 1.8 scaled score points, with an effect size of 0.14 stan- 
dard deviation. This effect size — 14 percent of a standard deviation between students in 
FAST-R and non-FAST-R schools — is notable but not statistically significant. 

The estimated impacts of FAST-R on the percentage of students scoring at a proficient 
or advanced level on the MCAS are 4.8 percentage points and 4.7 percentage points, respective- 
ly, for the first and second years after implementation.'^ These estimated impacts are not statisti- 
cally significant. The estimated impact for the SAT-9 Total Reading scaled score was 1.5 in the 
fall of 2006, but this impact was not statistically significant. The estimated impact for the 
SAT-9 Total Reading score in the fall of 2007 was -1.0 and was also not statistically signifi- 

^The average yearly trends for the percentage of students who scored proficient or advanced on the MCAS 
for both the representative FAST-R and the representative non-FAST-R schools during the five baseline years are 
positive (0.63 percentage point and 1.52 percentage points per year, respectively), suggesting that, in general, 
schools were showing some progress in the years before the FAST-R program was implemented. 
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cant.'° Overall, there are no statistically significant impacts of FAST-R on fourth-grade stu- 
dents’ MCAS ELA or SAT-9 Total Reading scores. 



Effects of the FAST-R Program on Students’ Ability to Make 
Inferences and Find Evidence 

Tables 4.3 and 4.4 display the estimated impacts of the FAST-R program on students’ 
ability to answer two types of questions found on the MCAS, making inferences from and find- 
ing evidence in reading passages." Table 4.3 displays the third-grade impacts. The first row 
shows FAST-R and non-FAST-R students’ average distance from the state mean on the third- 
grade MCAS reading test for the set of questions related to making inferences." Students in a 
representative FAST-R school, for instance, scored an average of 14.8 percentage points below 
the state average during the first school year after implementation. Students at a representative 
non-FAST-R school scored an average of 15.7 percentage points below the state average during 
the first year after the FAST-R program was implemented." The estimated impact of FAST-R 
on students’ ability to answer “making inferences’’ questions correctly during the first year after 
implementation is small and not statistically significant. The estimated impact on students’ 
ability to answer “finding evidence’’ questions correctly is negative and also not statistically 
significant. In the second year after implementation, the estimated impact on students’ ability to 
make inferences is larger, at 2.9 percentage points, with an effect size of 0.14 standard devia- 
tion. The estimated impact on students’ ability to find evidence is 1.8 percentage points with an 
effect size of 0.08 standard deviation. Although the estimated impacts on making inferences are 



"The average yearly trends for the SAT-9 Total Reading score for both the representative FAST-R and the 
representative non-FAST-R schools during the five baseline years are positive (0.68 and 0.92 scaled score points 
per year, respectively), suggesting that, in general, schools were showing some progress in the years before the 
FAST-R program was implemented. 

"The comparative inteimpted time series analysis of the questions on the MCAS that measure students’ abil- 
ity to make inferences and find evidence includes only three instead of five years of baseline scores in the trend 
(spring 2003 through spring 2005) because BPE has not classified the items on the spring 2001 and 2002 tests. 

"The average distance from the state mean on questions that were designed to measure students’ ability to 
making inferences and find evidence was calculated by subtracting the percentage of students who answered the 
question correctly across the state from each student’s score on a particular question and then calculating each 
student’s average score on all questions of a particular type. 

"The average yearly trend for the questions related to making inferences and finding evidence for both the 
representative FAST-R schools (-1.8 points per year for making inferences and -0.34 points for finding evidence) 
and the representative non-FAST-R schools (-1.08 points for making inferences and -0.08 for finding evidence) 
during the five baseline years are negative, suggesting that all schools in the study were showing some decline on 
these test scores before the FAST-R program was implemented compared with the state average. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 4.3 



Impacts on Third-Grade Students’ Performance on MCAS 
“Making Inferences” and “Finding Evidence” Questions 







Non- 


Estimated 


Estimated 

Impact 


P-Value for 
Estimated 


Outcome 


FAST-R 


FAST-R'* 


Impact Effect Size** 


Impact 


First vear after imolementation 












MCAS "making inferences" questions 
Deviation from trend 


-14.8 

2.2 


-15.7 

1.3 


1.0 


0.05 


0.628 


MCAS "finding evidence" questions 
Deviation from trend 


-14.5 

0.6 


-14.5 

1.4 


-0.8 


-0.04 


0.711 


Second vear after imnlementation 












MCAS "making inferences" questions 
Deviation from trend 


-13.1 

5.7 


-15.2 

2.8 


2.9 


0.14 


0.284 


MCAS "finding evidence" questions 
Deviation from trend 


-13.9 

1.6 


-16.1 

-0.2 


1.8 


0.08 


0.527 


Sample size" 






13,715 







SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000- 
2001 and 2006-2007 academic years. 

NOTES: The trend is constructed using three years of data before the FAST-R program was implemented (the 
2002-2003 through 2004-2005 academic years). The first year after implementation of FAST-R is the 2005-2006 
academic year, and the second year after implementation is the 2006-2007 academic year. 

MCAS = Massachusetts Comprehensive Assessment System. 

The scores for the MCAS questions on making inferences and finding evidence are the adjusted means during 
the years after implementation for the average, or "representative," school. To calculate these means, the trends 
and intercepts for each school were summed for all FAST-R and non-FAST-R schools, respectively. The trend 
and intercept were then used to calculate the predicted outcomes during each of the two school years after 
implementation. The deviation from the trend was added to these predicted outcomes to find the average student 
achievement scores. Estimated impacts are regression-adjusted for students' background characteristics and prior 
achievement. The deviation from trend for each follow-up year for the non-FAST-R schools is regression- 
adjusted. The deviation from trend for each follow-up year for the FAST-R schools is the impact added to the 
non-FAST-R schools' deviation from trend. 

"Making inferences" and "finding evidence" questions are measured as the average percentage point deviation 
from the state mean. 

A two-tailed t-test was applied to the estimated impact of the FAST-R program. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

“FAST-R schools were matched with multiple comparison schools. All comparison schools matched to a 
particular FAST-R school are weighted so that the total weight of all the matching non-FAST-R schools equals 
one. Since each comparison school could also he matched to several FAST-R schools, all the weights of a 
particular non-FAST-R school are summed. 

'’The estimated impact effect size was calculated by dividing the estimated impact by the standard deviation 
of outcomes for all students in the entire sample (by grade) during the 2002-2003 through 2004-2005 school 
years. 

“The sample size includes all third-grade students with reported MCAS scores in the FAST-R and 
non-FAST-R schools during the five years included in the analysis. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Table 4.4 



Impacts on Fourth-Grade Students’ Performance on MCAS 
“Making Inferences” and “Finding Evidence” Questions 

Estimated P-Value for 
Non- Estimated Impact Estimated 

Outcome FAST-R FAST-R° Impact Effect Size'’ Impact 



First year after implementation 



MCAS "making inferences" questions 


-9.7 


-11.2 








Deviation from trend 


0.9 


0.0 


0.8 


0.04 


0.708 


MCAS "finding evidence" questions 


-10.4 


-11.0 








Deviation from trend 


1.3 


1.9 


-0.6 


-0.03 


0.806 


Second year after imnlementation 












MCAS "making inferences" questions 


-10.9 


-13.5 








Deviation from trend 


-1.5 


-3.4 


2.0 


0.10 


0.501 


MCAS "finding evidence" questions 


-13.4 


-17.3 








Deviation from trend 


-1.5 


-4.2 


2.7 


0.12 


0.407 


Sample size‘s 






12,722 







SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000- 
2001 and 2006-2007 academic years. 

NOTES: The trend is constructed using three years of data before the FAST-R program was implemented (the 
2002-2003 through 2004-2005 academic years). The first year after implementation of FAST-R is the 2005-2006 
academic year, and the second year after implementation is the 2006-2007 academic year. 

MCAS = Massachusetts Comprehensive Assessment System. 

The scores for the MCAS questions on making inferences and finding evidence are the adjusted means during 
the years after implementation for the average, or "representative," school. To calculate these means, the trends 
and intercepts for each school were summed for all FAST-R and non-FAST-R schools, respectively. The trend and 
intercept were then used to calculate the predicted outcomes during each of the two years after implementation. 
The deviation from the trend was added to these predicted outcomes to find the average student achievement 
scores. Estimated impacts are regression-adjusted for students' background characteristics and prior achievement. 
The deviation from trend for each follow-up year for the non-FAST-R schools is regression-adjusted. The 
deviation from trend for each follow-up year for the FAST-R schools is the impact added to the non-FAST-R 
schools' deviation from trend. 

"Making inferences" and "finding evidence" questions are measured as the average percentage point deviation 
from the state mean. 

A two-tailed t-test was applied to the estimated impact of the FAST-R program. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

TAST-R schools were matched with multiple comparison schools. All comparison schools matched to a 
particular FAST-R school are weighted so that the total weight of all the matching non-FAST-R schools equals 
one. Since each comparison school could also he matched to several FAST-R schools, all the weights of a 
particular non-FAST-R school are summed. 

'’The estimated impact effect size was calculated by dividing the estimated impact by the standard deviation of 
outcomes for all students in the entire sample (by grade) during the 2002-2003 through 2004-2005 school years. 

'’The sample size includes all fourth-grade students with reported MCAS scores in the FAST-R and non-FAST- 
R schools during the five years included in the analysis. 
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larger than the estimated impacts on finding evidence, neither shows a statistically significant 
difference from zero. 



Table 4.4 displays the estimated impacts of FAST-R on fourth-grade students’ ability to 
make inferences from and find evidence in text.''' There are no statistically significant impacts 
for fourth-grade students on either of these types of questions. In the second year after imple- 
mentation, the magnitudes of the impacts for questions related to making inferences and finding 
evidence are similar, at 2.0 and 2.7, respectively, suggesting that the effects of FAST-R on 
fourth-grade students’ ability to answer these different types of questions are similar. 



Effects of the FAST-R Program on Students by Gender, 
Socioeconomic Status, Performance on a Pre-Program Reading 
Test, and Speciai Education Status 

The general findings for the analyses of student subgroups were similar to those of the 
overall impact analyses.'^ There were no statistically significant impacts found for students of 
lower or higher socioeconomic status or students performing below or above proficiency on a 
pre-program reading test. The impacts on the percentage of fourth-grade boys who scored at the 
proficient or advanced level on the MCAS are 8.7 percentage points during the first year after 
implementation and 7.5 percentage points during the second year after implementation. Both 
these impacts are statistically significant. However, the difference in impacts for girls and boys 
is not statistically significant, and, therefore, the higher impact for boys should be interpreted 
with caution. The impact on MCAS ELA total scores for third-graders receiving special educa- 
tion services is statistically significant during the second year after implementation, but the dif- 
ference in impacts on this measure for students receiving special education services versus stu- 
dents who are not receiving such services is not statistically significant and, again, should be 
viewed with caution. For fourth-grade students, there is a statistically significant impact on both 
the MCAS ELA score and the percentage of students who tested at a proficient or advanced 
level during the second year after implementation among students not receiving special educa- 
tion services. The difference between impacts on students receiving special education services 
and students who did not receive such services is not statistically significant for the MCAS ELA 
but is significant for the percentage of students performing at a proficient or advanced level. 

*'*The average yearly trend for the questions related to making inferences is positive for both the represent- 
ative FAST-R schools (1.09 points per year) and the representative non-FAST-R schools (1.15 points per year) 
during the five baseline years. The average yearly trend for questions on finding evidence is slightly negative 
(-0. 13 point for FAST-R students and -0.28 point for non-FAST-R students). 

'^Subgroup analyses at the school level were also attempted, including comparing schools where teachers re- 
ported spending more or less time with a FAST-R data coach and comparing schools that the Boston Plan for 
Excellence designated as high-implementers with schools that were still initiating their implementation at the time 
of the study. Neither of these cases had an adequate number of schools in each subgroup to conduct the analysis. 
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There is also a statistically significant impact on the SAT-9 Total Reading score during the 
second year after implementation for students receiving special education services. The differ- 
ence in impacts for this measure is also statistically significant. Still, the inconsistency of these 
impacts (showing up whether or not students receive special education services), and the small 
number of statistically significant impacts over all the analyses in this study suggest that these 
impacts should be interpreted cautiously.''’ 



'^See Appendix C for a more detailed description of the findings for each of the student subgroups. 
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Chapter 5 

Reflections and Conclusions 



The findings in the preceding chapters present a mixed picture of the Formative As- 
sessments of Student Thinking in Reading (FAST-R) intervention. The program was generally 
implemented as intended, and third- and fourth-grade teachers at schools participating in the 
initiative believed that they had benefited from the program. At least two-thirds of the teachers 
at these schools who responded to a survey administered as part of the MDRC evaluation re- 
ported that they had used FAST-R to assess students’ general reading comprehension and par- 
ticular reading skills, and that it had helped them understand how to use data to inform their 
teaching practices. 

On the other hand, the large majority of these teachers received only a few hours of 
FAST-R coaching over the course of the 2006-2007 school year. Moreover, their responses, as 
well as those of their counterparts at a set of matched comparison schools, make it clear that 
FAST-R was only one among many types of professional development activities that teachers 
in the Boston Public Schools system (BPS) received during the study period. BPS has made 
instructional improvement a district-wide goal and has publicly proclaimed that using data to 
identify student needs, improve teaching, and assess progress is one of “The Six Essentials for 
Whole-School Improvement.” It is not surprising, therefore, that teachers at the comparison 
schools also reported getting a good deal of professional development assistance, as well as 
spending time analyzing data, including data from formative assessments (albeit of a type dif- 
ferent from the formative assessments used in FAST-R). 

The central question posed by the impact analysis is whether students at FAST-R 
schools registered gains in reading scores that were significantly greater than the gains expe- 
rienced by similar students at the comparison schools, whose teachers were not exposed to 
FAST-R. The answer, in general, is that they did not. Differences in the achievement gains reg- 
istered by students at the two sets of schools were generally small and were not statistically sig- 
nificant (that is, they could have arisen as a result of chance). 

But if the numbers are relatively straightforward, what readers should make of the 
numbers is not. Complicating efforts at interpretation is the fact that the information about both 
the amount of professional development that teachers received and other implementation mat- 
ters is grounded in an imperfect source of data, the teacher survey. The response rate to the 
teacher survey was low — only 54 percent overall — and it is impossible to know whether the 
responses of teachers who completed the survey are an accurate representation of their counter- 



59 




parts who did not complete it.‘ To prompt further reflection on the issues, however, the follow- 
ing discussion assumes that the survey responses do present an accurate picture of conditions at 
the FAST-R and comparison schools, and considers the findings and several possible explana- 
tions in light of that assumption. These explanations must necessarily be regarded as specula- 
tive, however. 

Since the FAST-R program did not, in general, show an improvement over the status 
quo, one possible conclusion is that FAST-R is simply not an effective tool for raising student 
achievement — that is, that it would not work under any circumstances. But it is also possible 
that the specific conditions under which it operated in Boston undercut the likelihood that a po- 
tentially powerful intervention would show significant impacts. 

Thus, one plausible conclusion is that the intervention did not have positive impacts be- 
cause it was not intensive enough — that is, teachers did not receive a large enough “dose” of it 
for it to affect teaching and learning in meaningful ways. More than two-thirds of the teachers at 
FAST-R schools reported receiving five or fewer hours of FAST-R training over the course of 
the school year, and almost 10 percent of these teachers reported receiving no FAST-R profes- 
sional development at all. Teachers may not have had enough training in how to assess stu- 
dents’ learning needs and remediate instmction to address those needs.^ 

One way of testing this hypothesis would be to see whether FAST-R produced larger 
impacts on student achievement at those schools where teachers received a larger dose of the 
treatment. Unfortunately, such an analysis is infeasible because there were only three schools 
where teachers reported, on average, receiving 1 1 or more hours of FAST-R coaching. This 
number is too small to permit results to be reliable and generalizable. 

In any case, it should be noted that the Boston Plan for Excellence (BPE), the nonprofit 
organization that developed and mns FAST-R, would need to change its mode of operation in 
order to put in place a more intensive intervention at all schools. It would need to increase the 
number of instmctional data coaches in order to provide more professional development to 

*As noted in Chapter 2, the low response rate is probably attributable, at least in part, to the faet that con- 
traet negotiations had strained relationships between the teacher’s union and the district, and some teachers 
may have been unwilling to take on any efforts unrelated to their official duties. 

^It is also possible that the FAST-R program taught teachers how to analyze data and respond to students’ 
individual instmctional needs, but that teachers did not consistently put these lessons into practice in their class- 
rooms. This evaluation did not gather the systematic evidence that would allow researchers to examine this 
potential explanahon. While the study cannot shed light on the matter, the explanahon itself suggests that pro- 
fessional development is only the first step in a series of activities leading to instmctional improvement. Teach- 
ers must use what they have learned in planning group lessons and in responding to individual students’ learn- 
ing needs. School administrators, in their role as instmctional leaders, must go beyond arranging the delivery of 
professional development services to their teachers to monitor the effectiveness with which that professional 
development is translated into practice. 
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more teachers. It would also need to change its way of working with schools. As FAST-R oper- 
ated during the study period, each school could elect to receive as much or as little FAST-R 
coaching as it wanted, in line with its perceived needs and the availability of time for this par- 
ticular form of professional development. Schools signing up for FAST-R were not necessarily 
looking to add many more hours of professional development to their teachers’ schedules. If its 
aim were to increase the amount of professional development teachers receive, BPE would have 
to take a more proactive and forceful stance and leave less to the discretion of individual 
schools. 



Another plausible explanation for the absence of impacts is that while FAST-R may 
have increased student achievement over time, so did the other professional development efforts 
in place in the Boston Public Schools, which also included training in how to examine data. The 
survey findings indicate that teachers at the comparison schools received at least as much pro- 
fessional development of several different kinds as did teachers at the FAST-R schools. Teach- 
ers at the comparison schools also reported spending more time analyzing the results of the state 
reading assessment, and as much time studying formative assessment data, as did teachers at the 
FAST-R schools. In other words, the FAST-R “treatment” was not unique; while it may have 
represented a contribution to teachers’ knowledge and skills, this contribution was not very dif- 
ferent from what teachers would have received had they and their schools not participated in 
FAST-R. 

There is, however, another way of reading the data, one that suggests that FAST-R may 
have worked better than other training efforts and was, in fact, an “energy-efficient” and effec- 
tive form of professional development. In general, teachers at FAST-R schools reported spend- 
ing less time on professional development than their counterparts at the comparison schools — 
but students at the two groups of schools nonetheless experienced the same learning gains. 

Thus, the data presented in Table 3.2 indicate that for every type of professional devel- 
opment shown, a smaller proportion of non-FAST-R teachers reported receiving no profession- 
al development of that type, and a larger proportion reported receiving 1 1 or more hours of such 
professional development. For only one type of professional development — training specifical- 
ly related to the curriculum — is the difference between the two groups of teachers statistically 
significant, but the pattern is consistent across all types. The same pattern holds with respect to 
examination of the Massachusetts Comprehensive Assessment System (MCAS) data: Accord- 
ing to Table 3.5, non-FAST-R teachers were more likely to examine these data at all, and spent 
more hours doing so, than teachers in the FAST-R schools. 

If more professional development is associated with better instruction and (as several 
studies have shown) better insfruction is associated with better student outcomes, then achieve- 
ment gains at the non-FAST-R schools might be expected to surpass those at the FAST-R 
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schools.^ But they did not. It may be that the limited amount of professional development that 
BPE provided to the FAST-R schools was more useful to teachers than more diffuse profes- 
sional development efforts that took up more time. In judging the usefulness of FAST-R vis-a- 
vis other forms of professional development, it would be useful to have information about the 
costs of various professional development efforts. Such data were not collected as part of this 
study. 



This study has produced the best available data about FAST-R’s impacts, or the ab- 
sence of such impacts. But, as the preceding discussion indicates, the meaning of those data is 
very much open to discussion. Even if the correct interpretation were clear and incontrovertible, 
however, the study would not yield definitive evidence about the potential and policy relevance 
of the FAST-R approach; indeed, a single study in a single site is insufficient to establish 
whether any approach can be replicated with strong results and is therefore policy-relevant. It is 
the researchers’ hope that training in the use of formative assessments to guide instruction will 
be tested in many variations and in many places, in order to build a cumulative body of evi- 
dence about whether and how use of these assessments promotes better teaching and learning in 
America’s schools. 



^Brophy(1986). 
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Appendix A 

The Analytic Model Used in the FAST-R Impact Analysis 




The analytic model used to estimate impacts was stmctured to calculate each school’s 
deviation from its respective trend, constmcted using the five years of baseline data. It then 
compares the program schools’ deviations with the deviations of the comparison schools to es- 
timate impacts for the overall program, by grade.' The executed model is presented below: 

Yit = YPj * {Ft) + YFj * {Xij) + YSj + YSj * {Tt) + S/v + a + a 



Where: 






Yit 


= 


Student outcome for student, i, at time, t 


Pj 


= 


Dummy variable indicating whether school,y, is a program school 


Ft 


= 


Follow-up year at time, t 


Xij 


= 


Student-level baseline characteristics for student, i, in school,y 


Sj 


= 


School, j, fixed effect 


Tt 


= 


Trend at time, t 


V, 


= 


Random error term for cohort at time, t 




= 


Random error term for student, i 



As noted in the analytic model above, the impact estimation takes advantage of the stu- 
dent-level data and controls for variation in student characteristics through the use of covariates. 
Student-level covariates include age, gender, race/ethnicity, special education status. Limited 
English Proficient (LEP) or formerly EEP status, free or reduced-price lunch status, and Stan- 
ford Achievement Test, version 9 (SAT-9), Total Reading scaled score from the beginning of 
the third-grade school year. 

The coefficient on the program variable and its t-statistic provide a test of the statistical 
significance of the impact of the FAST-R program at follow-up time, t. This is essentially the 
difference between the deviation from the trend for the FAST-R schools and the deviation from 
the trend for the non-FAST-R schools. The error terms in this model account for the hierarchical 
nature of the data, with students nested within cohorts. 



'All comparison (non-FAST-R) schools matched to particular FAST-R schools were weighted equally so 
that the total weight of all the matching non-FAST-R schools equals one. Since each comparison school could 
also be matched to several FAST-R schools, all the weights of a particular non-FAST-R school are summed to 
create the weight given to each comparison school student. 
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Appendix B 

List of FAST-R and Non-FAST-R Schools 




Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Appendix Table B.l 

FAST-R and Non-FAST-R Schools: Third-Grade Sample 

FAST-R Schools Non-FAST-R Schools 



James J. Chittick Elementary School 
George H. Conley Elementary School 
David A. Ellis Elementary School 
Edward Everett Elementary School 
Dennis Haley Elementary School 
Harvard/Kent Elementary School 
Rafael Hernandez K-8 Elementary School 
Jackson/Mann K-8 School 
Mary Lyon K-8 School 
Joseph P. Manning Elementary School 
John Marshall Elementary School 
Samuel W. Mason Elementary School 
Mather Elementary School 
Mattahunt Elementary School 
Donald McKay K-8 School 
John D. Philbrick Elementary School 
Franklin D. Roosevelt K-8 School 
William E. Russell Elementary School 
Warren/Prescott K-8 School 
F. Lyman Winship Elementary School 



Sample size 



20 



Dante Alighieri Elementary School 
Phineas Bates Elementary School 
Ludwig van Beethoven Elementary School 
William E. Channing Elementary School 
Roger Clap Elementary School 
Paul A. Dever Elementary School 
Quincy E. Dickerman Elementary School 
John Eliot K-8 School 
Ralph Waldo Emerson Elementary School 
David G. Farragut Elementary School 
Emily A. Fifield Elementary School 
Elihu Greenwood Elementary School 
Henry Grew Elementary School 
James W. Hennigan Elementary School 
Henry L. Higginson Elementary School 
John P. Holland Elementary School 
Joseph J. Hurley K-8 School 
John F. Kennedy Elementary School 
Patrick J. Keimedy Elementary School 
Thomas J. Kenny Elementary School 
Joseph Lee Elementary School 
Wolfgang A. Mozart Elementary School 
Patrick O'Heam Elementary School 
William Ohrenberger Elementary School 
Pauline A. Shaw Elementary School 
Lucy Stone Elementary School 
Charles Sumner Elementary School 
Charles H. Taylor Elementary School 
William Monroe Trotter Elementary School 
Joseph P. Tynan Elementary School 
John Winthrop Elementary School 

31 



SOURCE; School names cited are from the Boston Public Schools online school directory. 

NOTE; FAST-R schools were matched with comparison non-FAST-R schools based on similarities in student 
demographics and achievement scores. Some comparison schools were matched to multiple FAST-R schools. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Appendix Table B.2 

FAST-R and Non-FAST-R Schools: Fourth-Grade Sample 

FAST-R Schools Non-FAST-R Schools 



James J. Chittick Elementary School 
George FI. Conley Elementary School 
David A. Ellis Elementary School 
Edward Everett Elementary School 
Dennis C. Flaley Elementary School 
Flarvard/Kent Elementary School 
Rafael Flernandez K-8 Elementary School 
Jackson/Mann K-8 School 
John Marshall Elementary School 
Samuel W. Mason Elementary School 
Mather Elementary School 
Mattahunt Elementary School 
Donald McKay K-8 School 
John D. Philbrick Elementary Sehool 
Franklin D. Roosevelt K-8 Sehool 
William E. Russell Elementary School 
Maurice J. Tobin K-8 School 
Warren/Prescott K-8 School 
F. Lyman Winship Elementary School 



Sample size 



19 



Phineas Bates Elementary School 
Ludwig van Beethoven Elementary School 
Roger Clap Elementary School 
Paul A. Dever Elementary School 
Quincy E. Dickerman Elementary School 
John Eliot K-8 School 
David G. Farragut Elementary School 
Emily A. Fifield Elementary School 
Elihu Greenwood Elementary School 
Sarah Greenwood K-8 School 
Flenry Grew Elementary School 
Nathan Flale Elementary School 
Alexander Flamilton Elementary School 
John P. Flolland Elementary School 
Joseph J. Flurley K-8 School 
John F. Kennedy Elementary School 
Thomas J. Kenny Elementary Sehool 
Joseph Lee Elementary School 
Wolfgang A. Mozart Elementary School 
William Ohrenberger Elementary School 
Michael J. Perkins Elementary School 
Oliver Flazard Perry K-8 School 
Pauline A. Shaw Elementary Sehool 
Lucy Stone Elementary School 
Charles Sumner Elementary School 
Charles FI. Taylor Elementary School 
William Monroe Trotter Elementary School 
Joseph P. Tynan Elementary School 
John Winthrop Elementary School 

29 



SOURCE; School names cited are from the Boston Public Schools online school directory. 

NOTE; FAST-R schools were matched with comparison non-FAST-R schools based on similarities in 
student demographics and achievement scores. Some comparison schools were matched to multiple FAST-R 
schools. 
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Appendix C 

Subgroup Analyses of the Effects of the 
FAST-R Program 




To better understand how the FAST-R program affected different groups of students, 
several subgroup analyses were conducted, including examining the effects of FAST-R on boys 
and girls, on students of lower and higher socioeconomic status as defined by whether or not a 
student was receiving free or reduced-price lunch, on students who performed below or above 
proficiency on a pre-program reading test, and on whether or not a student was receiving special 
education services. This appendix describes the findings of those analyses. 



Effects of the FAST-R Program on Students by Gender 

Tables C.l and C.2 display the effects of FAST-R on third-grade boys’ and third-grade 
girls’ achievement in reading. Although there are no statistically significant impacts on the 
Massachusetts Comprehensive Assessment System (MCAS) results for third-grade boys, the 
general pattern of estimated impacts for boys is positive. As shown in Table C.l, the estimated 
effect (deviation from trend) of FAST-R on third-grade boys’ distance from proficiency on the 
MCAS English Language Arts (ELA) test during the second year of implementation is 1 .3 raw 
score points, with an effect size of 0.16 standard deviation. As shown in Table C.2, the girls’ 
estimated impact for the MCAS distance from proficiency is negative in the first year of imple- 
mentation and positive but small in the second year after implementation (0.6 raw score point, 
with an effect size of 0.07 standard deviation). Still, a test of the difference in the impacts for 
boys and the impacts for girls shows that this difference is not statistically significant, and, 
therefore, it cannot be concluded that the impacts are different from each other.' 

Tables C.3 and C.4 display the effects of FAST-R on fourth-grade boys’ and fourth- 
grade girls’ reading achievement. In Table C.3, there is a statistically significant impact on the 
percentage of fourth-grade boys who scored at the proficient or advanced level on the MCAS 
during both the first and second year of implementation, with an estimated impact of 8.7 per- 
centage points during the first year after implementation and 7.5 percentage points during the 
second year after implementation. However, the difference in impacts for girls and boys is not 
statistically significant and, therefore, the higher impact on boys should be interpreted with cau- 
tion. Although not statistically significant, the estimated impact on the MCAS ELA score for 
fourth-grade girls during the second year after implementation, shown in Table C.4, is notable 
at 2.4 scaled score points, with an effect size of 0.19 standard deviation. The difference in im- 
pacts between boys and girls on this measure is also not statistically significant. 



'a t-test of the difference in impacts between boys and girls was done for each outcome. None of the out- 
comes showed statistically significant differences between the impacts for boys and girls, suggesting that cau- 
tion should be used when interpreting the difference in the magnitude of impacts between these groups. 
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Effects of the FAST-R Program on Students by Socioeconomic 
Status 

Approximately 80 percent of students in both the FAST-R and comparison schools 
were receiving free or reduced-price lunch, an indication of lower socioeconomic status. Tables 
C.5 and C.6 describe the effects of FAST-R on third-grade students’ achievement in reading by 
students’ socioeconomic status, and Tables C.7 and C.8 show the outcomes for fourth-grade 
students. In general, the patterns of effects for third- and fourth-grade students are similar to 
those of all students. The estimated impacts on the MCAS and the Stanford Achievement Test, 
version 9 (SAT-9) Total Reading scores for third-grade students receiving free and reduced- 
price lunch, as shown in Table C.5, are small and not statistically significant. The estimated im- 
pact on third-grade students’ distance from proficiency during the first year after implementa- 
tion of FAST-R is 0.0 raw score points while in the second year after implementation the esti- 
mated impact is 0.9 raw score points with an effect size of 0.11 standard deviation, but is not 
statistically significant. The estimated impacts on SAT-9 scaled scores for third-grade students 
receiving free or reduced price lunch are negative and not statistically significant. The estimated 
impacts on third-grade students not receiving free or reduced-price lunch, as shown in Table 
C.6, are negative and not statistically significant. 

The effects of FAST-R on low-income fourth-grade students’ MCAS scaled scores, as 
shown in Table C.7, are positive but are not statistically significant. During the first year after 
implementation, the estimated impact on the MCAS ELA test is 0.9 scaled score point, with an 
effect size of 0.07 standard deviation. The magnitude of the estimated impact during the second 
year is 2.3 scaled score points, with an effect size of 0. 1 8 and a p-value of 0. 1 06. The impacts of 
FAST-R for fourth-grade students not receiving free or reduced-price lunch, as shown in Table 
C.8, are, generally, small and not statistically significant. Although the magnitude of the impact 
on MCAS ELA score during the second year is notable for students of a lower socioeconomic 
status, the difference in impacts between the two groups is not statistically significant; thus, this 
finding should be viewed with caution. 



Effects of the FAST-R Program on Students by Baseline Reading 
Achievement 

Since the FAST-R program trains teachers to focus on individual student ability levels 
in reading, it is possible that FAST-R may be particularly successful with students at certain 
skill levels. Tables C.9 and C.IO display the effects of FAST-R on fourth-graders’ reading 
achievement by student performance on the SAT-9 test during the fall of their third-grade year. 



74 




before the FAST-R program was implemented.^ Table C.9 displays the effects of FAST-R on 
students who performed below the proficient level. A majority of the students in the study sam- 
ple were scoring at this level, and the impacts are similar to those for the full sample of students. 
Although there are no statistically significant impacts, the magnitude of the impact on the 
MCAS scaled score during the second year of implementation is notable at 1.9 scaled score 
points, with an effect size of 0.14 standard deviation. The effects on students performing at or 
above proficient, as shown in Table C.IO, are generally small and not statistically significant for 
MCAS, and negative and not statistically significant for SAT-9. The difference in impacts be- 
tween these two groups of students is not statistically significant for any of the outcomes. 



Effects of the FAST-R Program on Students by 
Special Education Status 

Tables C.ll and C.12 display the effects of FAST-R on third-grade students’ achieve- 
ment in reading by students’ special education status. The effect of FAST-R during the second 
year after implementation was statistically significant for students receiving special education 
services. A test of the difference in the impacts for students receiving special education services 
and those not receiving any special education services shows that this difference is not statistical- 
ly significant. Therefore, it cannot be concluded that the impacts are different from each other. 

Tables C.13 and C.14 display the effects of FAST-R on fourth-grade students’ 
achievement in reading by students’ special education status. During the second year after im- 
plementation, there are statistically significant impacts on students’ MCAS Total Reading 
scores (2.8 scaled score points, with an effect size of 0.22 standard deviation) and the percent- 
age of students scoring at the proficient or advanced level on the MCAS (9.4 percentage points) 
for students not receiving special education services. The difference in impacts for students re- 
ceiving and not receiving special education services is not statistically significant for the MCAS 
reading total measure but is significant for the percent of students scoring at a proficient or ad- 
vanced level. On the other hand, there is a statistically significant impact on the SAT-9 reading 
total score during the second year after implementation for students receiving special education 
services. The difference in impacts on this outcome is also statistically significant. Although 
these impacts are of a notable magnitude, they do not offer a coherent story. Since there are pos- 
itive and conflicting impacts on students both receiving and not receiving special education ser- 



^Only students who took the SAT-9 test in the fall of their third-grade year are included in this analysis, 
approximately 74 percent of the students in the sample. These students likely represent a less mobile group of 
students, since a principal reason students would not have taken the SAT-9 test in third grade is that they were 
not enrolled in the district at that time. A sensitivity test was conducted where the analysis was mn on only the 
students with SAT-9 pretest scores available. Patterns of impacts on this group of students were similar to those 
for the Ml sample. 
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vices, these impacts seem to be random and may be the consequence of testing error. The small 
number of statistically significant impacts of any magnitude over all the analyses in this study, 
coupled with the fact that the impacts on these subgroups are significant only at the 10 percent 
level (meaning there is a 10 percent chance that these findings are random and not due to actual 
differences in outcomes), suggests that these impacts should be interpreted with caution. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Appendix Table C,1 
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Appendix Table C.l (continued) 
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with SAT-9 scores reported in the fall of their fourth-grade year. 
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Appendix Table C,2 (continued) 
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with SAT-9 scores reported in the fall of their fourth-grade year. 




_2 

>■ 



H 

t /3 

<53 

SXI 

a 

■S 

Oi 



CXI 



O 



« - 



o> 

■O 

t /3 



o> 

s 



o> 

« 

s 

u 

o 



a 

H 

.^5 

■3 

fl 

V 

a 

a 



o "O 

<S £ 



^ W 

Oh 



”0 



W 



0> 

u 

o 

o 

c/i 

<Ji 

0> 

H 

&iD 

.s 

CQ 



o 

pa 

ci 

u 

o 



3 

o 

3 

O 



o 

3 

& 



D 


X) 

D 


Cd 










D 








c^-i 




pq 


”0 

D 


D 

Cd 


Cd 


9- 



s s 



w 



s 

o 

:z 



o 

o 

o 



'O 

(N 

<N 



oo o 

<N ' 
(N 



S -g 

lyj fl 



(L» 

Id £ 
9 o 

< 

-1 



^ > 
•< 0> 

U Q 

s 



o 

o 



m 



oo 

o 



< 

2 



r- 

oo 



m 'O 
'O 'O 



(N (N 
<N 



S 

0) 

”0 

3 

Ti 

o 



o 

e 

cd 

> 

”0 

cd 

o 

^ -o 

5= S 
D 0) 

‘o 

^ £ 
^ o 

*- P 

< g 

J .2 

^ > 
•< D 

U Q 



oo 

o 



oo 

(N 



o 

<N 

'O 



o m 

'O 



o 

o 

'ji 

”0 

0 ) 

3 

o 



0> 

”0 

cd 

feb 

-fi 

3 

^ -o 

^ § 

^ a 
s> § 
- * 
3 s 
o 



0 
H 
o^ 

1 

H 

< 

GO 



u 

H 

3 

S 

© 



C/J 



m 

CN 

o 



'O (N 
(N ' 
CN 



OO 

CN 

(N 



S -g 

tyj S 



o ° 

P O 

< g 

J .2 

^ > 
■< 0> 

U Q 



o 

oo 

o 

o 



<N 

r- 



0\ 

o 



< 

2 



o 
oo 'O 



(N 

<N 



e 

fD 

”0 

3 

fy) 

O 

”0 

D 

O 

G 

cd 

> 

”0 

cd 

;-( 

O 

^ "O 
G C 

a> fD 



o 

^ p 

*- p 

CXcfe 
< 



cd 



-1 

tq 

^ > 
■< fD 

U Q 

s 



(N 

'O 



r- o 

<N 

'O 



o 

CO 

'Ji 

”0 

ID 

3 

D 



D 

”0 

cd 

& 

-C 



^ ”0 
bS ^ 
s 

"S i 

* 



D 



Cd 

0 
H 
o^ 

1 

H 

< 

c/:) 



-o 

D 



s 

o 

D 



O 

O 



(N 

tn 

oC 



cd 

GO 



81 




Appendix Table C,3 (continued) 
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with SAT-9 scores reported in the fall of their fifth-grade year. 
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Appendix Table C.4 (continued) 
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(continued) 




SOURCES: MDRC calculations from individual student records from Boston Public Schools between the 2000-2001 and 2006-2007 
academic years. 
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MCAS scores in fourth grade. The sample size for the SAT-9 Total Reading is 2,950 students and includes all fourth-grade students with 
SAT-9 scores reported in the fall of their fifth-grade year. 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Appendix Table C.12 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Appendix Table C.13 
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Appendix Table C,13 (continued) 
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Formative Assessments of Student Thinking in Reading (FAST-R) Evaluation 

Appendix Table C.14 
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Appendix Table C ,14 (continued) 
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Appendix D 

Sample of FAST-R Assessment 
Student and Teacher Materials 




Appendix D features sample FAST-R assessment materials created by the Boston Plan 
for Excellence (BPE). 

The sample FAST-R assessment shown on the following pages was given to third- and 
fourth-grade students in the Boston Public Schools. The assessment passage is entitled “Apart- 
ment Building,” by Eileen Spinelli. After reading the story, students answer 1 1 questions based 
on the “drawing inferences” and “finding evidence” reading skills. 

The assessment materials also include a FAST-R teacher’s guide explaining the scoring 
process BPE used to grade students’ assessments. Teachers can see the reading skills and vocabu- 
lary highlighted in the text. The guide also provides teachers with ideas for follow-up lessons con- 
nected to the exercise. 

Teachers can tell whether a question is assessing the “making inferences” or “finding 
evidence” reading skills in the FAST-R assessment. In addition, all wrong answers to a question 
are coded as OOPl, indicating answers that are hue but irrelevant based on the text; OOP2, in- 
dicating “misread” answers based on a misunderstanding of the reading text; or OOB, which 
represents “out of bounds” answers that are not based at all on the text. 

Following the teacher’s guide in this appendix is a sample of a FAST-R assessment da- 
ta report that teachers receive outlining their students’ performance on the assessment. Teachers 
can see how students performed overall as a class, individually, and by subgroup (such as race, 
ethnicity, gender, limited English proficient, and special education). 

Finally, a FAST-R “Student Performance Summary” explains the various types of 
questions that measure students’ ability to make inferences and find evidence in reading passag- 
es. Teachers can pinpoint the strengths and weaknesses of their students with this enhanced lev- 
el of question-item analysis. 
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FAST-R 





+< !! 



Formative Assessments of Student Thinking in Reading 

Name "Apartment Building” • Fiction 



Date Teacher/Class 




Apartment Building 

by Eileen Spinelli 



In this story, a boy named Sam solves a 
problem in a very interesting way. Read 
this story to find out what his problem is, 
and how he solves it. Answer the questions 
that follow. 



1 Sam wanted a pet. A little one. A big one. One 
that flew or walked. Any kind of pet at all. 

2 But Mrs. Cabot, the landlady, said: “NO 
PETS!” And that was that. 

3 Sam’s mom and dad bought him fuzzy stuffed 
hears and a kite in the shape of a kitten and a plas- 
tic parrot on a stick. But he wanted a live pet. And 
that was that. 

4 Sam tried to change Mrs. Cahot’s mind. He 
told her he would get a quiet pet. He told her he 
would keep it clean. He told her he would not let 
the pet scratch the woodwork or jump through the 
hedges. 

^ But Mrs. Cahot looked Sam square in the eye 
and said: “NO PETS!” 

g One day Sam heard Mrs. Cahot 
screaming in the hallway. He 
rushed over. “What’s wrong, Mrs. 

Cahot?” 

7 “A mouse!” she shrieked. “I saw 
a mouse!” 



“I thought you had a rule, Mrs. Cahot. No pets 8 
allowed.” 

“It wasn’t a pet mouse,” the landlady squawked. 9 
“It was a plain old terrorize-the-huilding type of 
mouse.” 

Sam grinned. “What you need, Mrs. Cahot, is a 10 
cat. 

While it was true that Mrs. Cahot hated cats, n 
there was something she hated even more - mice! 
And so that very day, Mrs. Cahot went to the ani- 
mal shelter and found herself a cat. 

The next time Sam saw Mrs. Cahot, he said, “I 12 
see you have a cat.” 

“Yes indeed,” she replied. 

“Does that mean I can get a pet?” Sam asked. 14 

“No, it does not!” snapped Mrs. Cahot. “If I 15 
let you get a pet. I’d have to let everyone get a pet. 

Then I’d he running a zoo and not 
an apartment building!” 

Another day Sam was coming 16 
home from school. Police cars sur- 
rounded the apartment building. 
“What happened, Mrs. Cabot?” he 
asked. 



Spotlight On: Eileen Spinelli 

Eileen Spinelli has written over 20 
books, including picture books and 
collections of poems. She is the author 
of the Lizzy Logan series. She has six 
children and thirteen grandchildren. 
Her husband, Jerry Spinelli, is also a 
writer. His work includes Maniac Magee. 



FAST-R: Formative Assessments of Student Thinking in Reading. The passage text by Eileen Spinelli is from Highlights for Children. Copyright © 1 990 by Eileen Spinelli. Some guestions 
were drawn or adapted from the G3 MCAS Spring 2001 test. All other materials are Copyright 2007 by the Boston Plan for Excellence. 



36 



17 “I was robbed!” she cried. “They took my radio 
and my penny bank and my entire collection of salt 
and pepper shakers!” 

18 “That’s too bad,” said Sam, shaking his head. 
“What you need is a good watchdog.” 

19 While it was true that Mrs. Cabot hated dogs, 
there was something she hated even more - robbers! 
And so that very day, Mrs. Cabot went to the ken- 
nel and bought herself a dog. 

70 A week later Sam found Mrs. Cabot dusting the 
stairs. There were tears in her eyes. 

21 “You look sad,” said Sam. 

22 'T am sad,” replied Mrs. Cabot. “My very best 
friend in all the world is moving away.” 

23 “I’m sorry to hear that,” said Sam. 

24 “She and I would talk for hours,” sniffled Mrs. 
Cabot. 

25 “Can’t you talk to Mr. Cabot?” asked Sam. 

26 Mrs. Cabot shook her head. “Mr. Cabot likes to 
read the newspaper. He likes to watch TW. He likes 
to build shelves for his wrench collection. But he 
doesn’t like to talk.” 

27 “What you need, Mrs. Cabot, is a parrot.” 

28 Mrs. Cabot dabbed at her eyes with the hem of 
her apron. “A parrot?” 

79 Sam nodded. “Parrots love to talk.” 

30 While it was true that Mrs. Cabot hated birds, 
there was something she hated even more - not 
having anyone to talk with. And so that very day, 
Mrs. Cabot went to The Exotic Bird Shop and 
bought herself a parrot that talked all the way 
home. 

31 In spring the grass grew green and thick and tall. 
Mrs. Cabot tried to mow the lawn, but every time 
she pushed the mower, she sneezed. 

32 “Bless you,” said Sam. 

33 “Thank you,” said Mrs. Cabot, between sneezes. 

34 “Do you have a cold?” Sam asked. 

35 Mrs. Cabot blew her nose. Sneezed. Pushed the 
mower. Then sneezed again. “No. I have allergies.” 



“You shouldn’t be mowing grass, then,” said 
Sam. 

“I know that,” replied Mrs. Cabot, sneezing. 37 
“But Mr. Cabot hurt his back. So he can’t mow the 
grass.” 

“Maybe I could,” offered Sam. 38 

“That’s kind of you,” said Mrs. Cabot. “But 39 
you’re too small.” 

Sam smiled. “I know just what you need.” 40 

“What’s that?” asked Mrs. Cabot. 41 

“A goat!” Sam laughed. “A goat will eat every 42 
bit of grass. You’ll never have to mow again.” 

While it was true that Mrs. Cabot hated goats, 43 
there was something she hated even more - sneez- 
ing! And so that very day, she drove to a farm and 
came back with a goat. 




A month later Sam found Mrs. Cabot hammer- 
ing a For Sale sign into the front lawn. 

“Are you selling the building?” asked Sam. 45 

“I don’t want to,” sighed Mrs. Cabot. “But I’m 46 
so busy taking care of the cat and the dog and 
the parrot and the goat that I don’t have time for 
anything else. The laundry room is full of cobwebs. 
The stairs are full of dust. And as for the hedges 
- well, see for yourself.” 

Sam patted Mrs. Cabot on the shoulder. “What 47 
you need is a pet-sitter.” 

Mrs. Cabot stopped hammering. “But who on 48 
earth would take care of all those pets?” 

Sam’s grin was as wide as a wheelbarrow. “I 49 
know just the person,” he said. “And he even lives 
in the building!” 



FAST-R 




Formative Assessments of Student Thinking in Reading 

Name "Apartment Building” • Fiction 



Date Teacher/Class 



Fill in the circle for the best answer to each question. 

Apartment Building 

1, The introduction says Sam has a problem to solve. What was that problem? 

He wanted a pet but wasn’t allowed to have one. 

He had allergies whenever someone cut the grass. 

Mrs. Cabot was selling the apartment building and his family had to move. 

He was very busy taking care of all Mrs. Cabot’s pets. 

2, According to the passage, the MOST LIKELY reason that Mrs, Cabot did not allow pets was 

that 

she didn’t like animals, 
she was allergic to animals. 

@ she wanted to run a zoo. 

@1 Mr. Cabot was afraid of animals. 

3, According to paragraph 4, how did Sam try to change Mrs, Cabot’s mind about letting him 

have a pet? 

He said he would buy her a kite or a stuffed bear. 

He offered to take out the garbage for a month. 

@ He told her that pets could help solve her problems. 

@) He promised his pet would be quiet and clean. 

4, Why did Sam grin when he suggested Mrs, Cabot buy a cat? 

He thought it was funny that Mrs. Cabot hated mice. 

@ He hoped Mrs. Cabot would get a cat even though she had a rule against pets. 

© He liked mice. 

He knew that Mrs. Cabot liked cats. 

5, According to the passage, why did Sam think that Mrs, Cabot should get a goat? 

@ to chase away robbers 

(b) so she could start a petting zoo next to the apartment building 
© so she wouldn’t have to mow the lawn 

(5) because she was allergic to cats Continue on the next page 



FAST-R: Formative Assessments of Student Thinking in Reading. The passage text by Eiieen Spineiii is from Flighiights for Chiidren. Copyright © 1 990 by Eileen Spineili. Some guestions 
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6, What did all of Mrs, Cabot’s pets have in common? 

She bought all of them from pet stores. 

She ehose all of them to help her in some way. 

@ They were all pets Sam did not want. 

@ They were all good at doing tricks. 

7, What was the MAIN reason Sam kept suggesting animals for Mrs, Cabot to get? 

He wanted Mrs. Cabot to have a pet instead of him. 

He liked helping Mrs. Cabot. 

@ He hoped Mrs. Cabot would let him get a pet too. 

@ He wished that Mrs. Cabot would move away. 

8, Why did Mrs, Cabot want to sell the apartment building? 

She did not want to take care of all her pets anymore. 

@ She was afraid of robbers. 

(@ She thought she could make money from selling the building. 

@) She could not take care of both the animals and the apartment building. 

9, Who did Sam think would be a good pet-sitter for Mrs, Cabot? 

himself 

(B) his parents 

(C) Mr. Cabot 

@ the man who walks his dog past the building 

10, What is the MAIN IDEA of this story? 

@ Sometimes you can get what you want by helping others. 

Sometimes you can find pets at an animal shelter. 

Sometimes there’s not enough work to do around an apartment building. 

@ Sometimes you can meet new people near your home. 

Open Response Question: 

Explain Sam’s problem and how he solves it. Use important information from the selection in your 
answer. 



FAST-R: Formative Assessments of Student Thinking in Reading. The passage text by Eileen Spinelli is from Flighlights for Children. Copyright © 1 990 by Eileen Spinelli. Some guestions 
were drawn or adapted from the G3 MCAS Spring 2001 test. All other materials are Copyright 2007 by the Boston Plan for Excellence. 



Teachers: Please duplicate and use this answer sheet only for students for whom you did not receive a pre-printed answer sheet! 



FAST-R Answer Sheet 

Name 

Date 

Passage Title 

Completely fill the circle 
for the correct answer. 
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School 

Grade Class 

Teacher Name 

Write your answer to the open response prompt in the lined space below 
if your teacher directs you to do so. 



OFFICE USE ONLY 

RESEARCH: (f) 0 

OPEN RESPONSE: ©©00 



Teacher Guide for FAST-R Passage: 

FAST-R: Formative Assessments of Student Thinking in Reading 



Apartment Building 
Realistic Fiction 



At a Glance 

Approximate ■ 

Grade Range: 2 3456789 10 11 

Genre: Realistic Fiction 

Topic: By convincing his landlady that various animals can 

help solve her problems, Sam winds up getting to “pet- 
sit” despite the landlady’s ban on pets. 

Author: Eileen Spinelli 
Source: Highlights for Children 



Difficulty Index: Considerate Challenging 

Structure: 

Purpose: 

Richness: 

Relationships: 

Vocabulary: 

Style: 

Lexile Measure: 61 OL 



In this story, a boy named Sam solves a problem in a very 

interesting way. Read this story to find out what his problem is, 
and how he solves it. Answer the questions that follow. 

Apartment Building 

by Eileen Spinelli 

1 Sam wanted a pet. A little one. A big one. One that flew or 
walked. Any kind of pet at all. 

2 But Mrs. Cabot, the landlady, said: “NO PETS!” And that was 
that. 

3 Sam’s mom and dad bought him fuzzy stuffed bears and a kite in 
the shape of a kitten and a plastic parrot on a stick. But he wanted a 
live pet. And that was that. 

4 Sam tried to change Mrs. Cabot’s mind. He told her he would get 
a quiet pet. He told her he would keep it clean. He told her he would 
not let the pet scratch the woodwork or jump through the hedges. 

5 But Mrs. Cabot looked Sam square in the eye and said: “NO 
PETS!” 



Structure: Sam's “problem,” which is alluded to in the 
introductory blurb, is that he wants a pet but his landlady 
prohibits them. After Sam tries to persuade her to relent 
(T|4), Mrs. Cabot encounters a series of problems that 
Sam suggests could be solved by an animal. 

Repetitive text elements add predictability and make the 
story particularly considerate for struggling readers. The 
story ends with a final twist that solves Sam’s problem by 
solving Mrs. Cabot’s. 

See especially: Questions 1 , 4, 6, 10 



6 One day Sam heard Mrs. Cabot screaming in the hallway. He 
rushed over. “What’s wrong, Mrs. Cabot?” 

7 “A mouse!” she shrieked. “I saw a mouse!” 



8 “I thought you had a rule, Mrs. Cabot. No pets allowed.” 



9 “It wasn’t a pet mouse,” the landlady squawked. “It was a plain old 
terrorize-the-building type of mouse.” 

10 Sam grinned. “What you need, Mrs. Cabot, is a cat.” 

1 1 While it was true that Mrs. Cabot hated cats, there was something 
she hated even more - mice! And so that very day, Mrs. Cabot went to 
the animal shelter and found herself a cat. 

12 The next time Sam saw Mrs. Cabot, he said, “I see you have a cat.” 

1 3 “Yes indeed,” she replied. 

14 “Does that mean I can get a pet?” Sam asked. 

15 “No, it does not!” snapped Mrs. Cabot. “If I let you get a pet. I’d 
have to let everyone get a pet. Then I’d be running a zoo and not an 
apartment building!” 



■< 



Relationships: Sam identifies Mrs. Cabot's needs and 
suggests animals as a solution, thereby hoping to meet 
his own “need” as well. 

See especially: Questions 4, 6, 7, 10 



1 6 Another day Sam was coming home from school. Police cars sur- 
rounded the apartment building. “What happened, Mrs. Cabot?” he 
asked. 



17 “I was robbed!” she cried. “Uey took my radio and my penny Continued on next page 

bank and my enure collection ot salt and pepper shakers! 



FAST-R: Formative Assessments in Student Thinking in Reading. The passage text by Eileen Spinelli is from Highlights for Children. Copyright © 1 990 by Eileen Spinelli. Some questions were 
drawn or adapted from the G3 MCAS Spring 2001 test. All other materials are Copyright 2007 by the Boston Plan for Excellence. 



1 8 “That’s too bad,” said Sam, shaking his head. “What you need is a 
good watchdog.” 

19 While it was true that Mrs. Cabot hated dogs, there was something 
she hated even more - robbers! And so that very day, Mrs. Cabot went 
to the kennel and bought herself a dog. 

20 A week later Sam found Mrs. Cabot dusting the stairs. There were 
tears in her eyes. 

21 “You look sad,” said Sam. 

22 “I am sad,” replied Mrs. Cabot. “My very best friend in all the 
world is moving away.” 

23 “Fm sorry to hear that,” said Sam. 

24 “She and I would talk for hours,” sniffled Mrs. Cabot. 

25 “Can’t you talk to Mr. Cabot?” asked Sam. 

26 Mrs. Cabot shook her head. “Mr. Cabot likes to read the news- 
paper. He likes to watch TV. He likes to build shelves for his wrench 
collection. But he doesn’t like to talk.” 




27 “What you need, Mrs. Cabot, is a parrot.” 

28 Mrs. Cabot dabbed at her eyes with the hem of her apron. “A par- 
rot?” 



29 Sam nodded. “Parrots love to talk.” 

30 While it was true that Mrs. Cabot hated birds, there was some- 
thing she hated even more — not having anyone to talk with. And so 
that very day, Mrs. Cabot went to The Exotic Bird Shop and bought 
herself a parrot that talked all the way home. 

31 In spring the grass grew green and thick and tall. Mrs. Cabot tried 
to mow the lawn. But every time she pushed the mower, she sneezed. 

32 “Bless you,” said Sam. 

33 “Thank you,” said Mrs. Cabot, between sneezes. 

34 “Do you have a cold?” Sam asked. 

35 Mrs. Cabot blew her nose. Sneezed. Pushed the mower. Then 
sneezed again. “No. I have allergies.” 

36 “You shouldn’t be mowing grass, then,” said Sam. 

37 “I know that,” replied Mrs. Cabot, sneezing. “But Mr. Cabot hurt 
his back. So he can’t mow the grass.” 

38 “Maybe I could,” offered Sam. 

39 “That’s kind of you,” said Mrs. Cabot. “But you’re too small.” 

40 Sam smiled. “I know just what you need.” 

41 “What’s that?” asked Mrs. Cabot. 

42 “A goat!” Sam laughed. “A goat will eat every bit of grass. You’ll 
never have to mow again.” 

43 While it was true that Mrs. Cabot hated goats, there was some- 
thing she hated even more - sneezing! And so that very day, she drove 
to a farm and came back with a goat. 

44 A month later Sam found Mrs. Cabot hammering a For Sale sign 
into the front lawn. 



Spotlight On: Eileen Spinelli 

Eileen Spinelli has written over 20 books, including 
picture books and collections of poems. She is the 
author of the Lizzy Logan series. She has six children 
and thirteen grandchildren. Her husband, Jerry 
Spinelli, is also a writer. His work includes Maniac 
Magee. 

Open Response Prompt: 

Explain Sam’s problem and how he solves it. Use im- 
portant information from the selection in your answer. 



45 “Are you selling the building?” asked Sam. 

46 “I don’t want to,” sighed Mrs. Cabot. “But I’m so busy taking care 
of the cat and the dog and the parrot and the goat that I don’t have 
time for anything else. The laundry room is full of cobwebs. The stairs 
are full of dust. And as for the hedges - well, see for yourself” 

47 Sam patted Mrs. Cabot on the shoulder. “What you need is a pet- 
sitter.” 

48 Mrs. Cabot stopped hammering. “But who on earth would take 
care of all those pets?” 

49 Sam’s grin was as wide as a wheelbarrow. “I know just the person,” 
he said. “And he even lives in the building!” 



Structure: The final twist solves Sam’s "problem” while 
once again solving one of Mrs. Cabot’s problems. The 
text leaves implicit just who Sam intends to suggest as 
a pet-sitter, but his overall approach to the problem- 
solution pattern, as well as his “grin as wide as a 
wheelbarrow” provide strong clues. 

See especially: Question 9 
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Teacher Guide for FAST-R Passage: Apartment Building 

FAST-R: Formative Assessments of Student Thinking in Reading j$tic FictiOH 

The annotated answer key below highlights common reasons students might choose each answer, and the sidebar gives more 
insight into the question types, to help you understand patterns of student responses. Always make time to follow up with students 
in conferences or small groups to probe their thinking, teach in response to patterns, and help them apply effective reading and 
thinking strategies to their everyday reading. 

Note: You may find it helpful to refer to the “Types and Levels of Thinking Assessed on FAST-R” sheet from your teacher 
resource folder as you examine your students' responses. The icon in the right-hand column, below, corresponds to that sheet's 
more detailed explanations of the kinds of thinking each type of question asks of readers. 



1. The introduction says Sam has a problem to solve. What was that problem? MI: Determine implicit 

A. He wanted a pet but wasn’t allowed to have one. { T|2) meaning from ideas in 

B. He had allergies whenever someone cut the grass. (OOP2, ^35) context 

C. Mrs. Cabot was selling the apartment building and his family had to move. 

(OOP2, H44-46) 

D. He was very busy taking care of all Mrs. Cabot’s pets. (OOP2, 1|48) 




2. According to the passage, the MOST LIKELY reason that Mrs. Cabot did not 
allow pets was that 

A. she didn’t like animals. — can be inferred from the fact that each time Sam 
suggests an animal to solve her problem, it says Mrs. Cabot “hated” the animal 

B. she didn’t want too many animals in the building. — (OOPl) at the start it 
says he didn’t want ANY animals in her building 

C. she wanted to run a zoo. — (OOP2) Mrs. Cabot snapped, “Then Fd be run- 
ning a zoo and not an apartment building, ” implying disfavor at the idea 

D. Mr. Cabot was afraid of animals. — (OOB) no evidence suggests this 



MI: Determine implicit 
meaning by understanding 
the organization of 
information in the text 




3. According to paragraph 4, how did Sam try to change Mrs. Cabot’s mind about FE: Identify evidence 

letting him have a pet? explicitly stated in the text 

A. He said he would buy her a kite or a stuffed bear. — (OOP2, T|3) Sam’s 
parents bought him a kite and a fuzzy stuffed bear 

B. He offered to take out the garbage for a month. — (OOB) a perstiasive 
tactic students may have tried on adults in their lives, but not in the text 

C. He told her that pets could help solve her problems. — (OOPl, true, but 
that did not persuade her to let him get a pet of his own [see 5-16 ‘Does that 
mean I can get a pet?” “No, it does not!”^ 

Y D. He promised his pet would be quiet and clean. — (T|4) 




4. Why did Sam grin when he suggested Mrs. Cabot buy a cat? MI: Determine implicit 

A. He thought it was funny that Mrs. Cabot hated mice. — (OOB) a grin meaning from words in 

may signal humor, but Sam “rushed over” out of concern when she screamed context 

Y B. He hoped Mrs. Cabot would get a cat in spite of her rule against pets. 

— Qxin be inferred from his questions to Mrs. Cabot about getting a pet 

C. He liked mice. — (OOPl) Probably true, since he wanted “any kind of pet at 
all, ” but not the reason he grinned 

D. He knew that Mrs. Cabot liked cats. — (OOP2) Sam himself liked cats, but 
Mrs. Cabot “hated cats” 
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5. According to the passage, why did Sam think Mrs. Cahot should get a goat? 

A. to chase away robbers — (OOP2) he suggested a dog to chase robbers 

B. so she could start a petting zoo next to the apartment building — (OOB) 
goats are common in petting zoos, but no evidence supports this choice 

C. so she wouldn’t have to mow the lawn — f ^ 42) 

D. to cure her allergies — (OOPl) the goat wont “cure” her grass allergies, but 
would help alleviate them by reducing her need to mow the lawn 



FE: Recognize the explicit 
meaning from varied 
wording in the text 




6. What did all of Mrs. Cahot’s pets have in common? MI: Determine implicit 

A. She bought all of them from pet stores. — (OOPl) only the bird came from meaning from words in 

a pet store; the others came from an animal shelter, kennel, or farm context 

'd B. She chose all of them to help her in some way. — can be inferred from 

evidence throughout the passage that each pet was bought for a specific purpose 

C. They were all pets Sam did not want. — (OOP2) clearly not true, since he 
eagerly ojfered to care for them at the end 

D. They were all good at doing tricks. — (OOB) though they were intended to 
help her, it was not by doing tricks 




7. What was the MAIN reason Sam kept suggesting animals for Mrs. Cahot to get? 

A. He wanted Mrs. Cabot to have a pet instead of him. — (OOP2) Sam 
hoped he and Mrs. Cabot could have pets 

B. He liked helping Mrs. Cabot. — (OOPl) Sam ojfered to mow the lawn and 
pet-sit, but helpfulness is not the main reason he wants her to get pets 

'd C. He hoped Mrs. Cahot would let him get a pet too. - (T| 14) 

D. He wished that Mrs. Cabot would move away. — (OOB) 



MI: Determine implicit 
meaning by understanding 
the organization of 
information in the text 




8. Why did Mrs. Cahot want to sell the apartment building? FE: Recognize the explicit 

A. She did not want to take care of all her pets anymore. — (OOP2) she’s too meaning from varied 

busy to care for animals and the building, but is keeping the animals wording in the text 

B. She was afraid of robbers. — (OOPl) she was afraid of robbers, but that’s 
why she got a dog, not why she wants to sell the building 

C. She thought she could make money from selling the building. — (OOB) 
logical (especially in Boston’s real estate market!) but not implied 

'd D. She could not take care of hoth the animals and the apartment building. 

— (H46) 




9. Who did Sam think would be a good pet-sitter for Mrs. Cabot? 
'd A. himself 

B. his parents (OOP2) 

C. Mr. Cabot (OOP2) 

D. the man who walks his dog past the building (OOB) 



MI: Determine implicit 
meaning from ideas in 
context 




10. What is the MAIN IDEA of this story? 

■d A. Sometimes you can get what you want by helping others. — by helping 
Mrs. Cabot solve her various problems, Sam gets what he wants 

B. Sometimes you can find pets at an animal shelter. — (OOPl) the cat was 
found at an animal shelter, but this is not the main idea of the story 

C. Sometimes there’s not enough work to do around an apartment building. 
— (OOP2) there was too much work to be done 

D. Sometimes you can meet new people near your home. — (OOB) this may 
sound reasonable to students, but is not related to the story 



MI: Determine a singular 
meaning from the total text 
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Teacher Guide for FAST-R Passage: Apartment Building 

FAST-R: Formative Assessments of Student Thinking in Reading R0ji|jstic FictiOH 



What levels and types of thinking are assessed by questions on this FAST-R question set? 



O w 
(0 O 




“Making inference” questions ask readers 
to determine implicit meaning ... 











...by 
incorporat- 
ing literary 
knowledge. 



... from particular words, phrases, or Ideas in context. 



Q1,4,6,9 





“Finding evidence” 
questions ask readers 
to determine 
explicit 
meaning ... 



... by recognizing evidence explicitly stated at multiple locations 
or with varied wording in the text. 

Q5,8 
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FAST-R Student Performance Summary 

These graphs show how many students selected each answer 
choice, while the annotated answer key highlights types of wrong 
answers, where answers were found in the text, and what 
thinking may have led to that answer choice. 

The data can help you hypothesize about your students' reading 
and thinking, but talking with your students will confirm or 
contradict your hypotheses and help you plan mini-lessons, 
conferring, and guided reading instruction in response to your 
students' strengths and needs. 



This analysis represents the performance of: 

School Sample ES Grade: 04 Class/Sec: 08 Question Set: G4F1 



Key Correct answer 

OOPl "near miss" answer; true 
based on text, but irrelevant 

00P2 "mis-read" answer; based on 
misunderstanding of the text 

OOB answer; not based on text; 
plausible from prior knowledge 

No answer 



1. How did Sam try to change Mrs. Cabot's mind about letting him have a pet? 



1 . Rnding Evidence 



A. He said he would buy her a kite or a stuff ed bear. — (00P2, 13) Sam's parents bought 
him a kite and a fuzzy stuffed bear 

B. He offered to take out the garbage for a month. — (OOB) a persuasive tactic students 
may have tried on adults in their lives, but not in the text 

C. He told her that pets could help solve her problems. — (OOPl) true, but did not 
persuade her (see 115-16 "Does that mean I can get a pet?" "No, it does not!" 

D. He promised his pet would be quiet and clean. — (14) 



2. Why did Mrs. Cabot get a parrot? 

A. She liked birds. — (OOB) sounds reasonable, not supported by the text 

B. She got a plastic parrot for decoration. — (00P2) in 13, Sam's parents bought him a 
plastic parrot; not the reason she got a real parrot 

C. So that Sam could help take care of it. — (OOPl) at the end, Sam offers to help care for 
the pets, but that's not why Mrs. Cabot gets a parrot 

y D. She didn't have anyone to talk with. — found about two-thirds of the way through the 
passage 



3. All of Mrs. Cabot's pets were the SAME in what way? 

A. She bought all of them from pet stores. — (OOPl) only the bird came from a pet store; 
the others came from a shelter, kennel, or farm 

y B. She chose all of them to help her in some way. — can be inferred from evidence 
throughout the passage that each pet was bought for a specific purpose 

C. They were all pets Sam did not want. — (00P2) clearly not true; he eagerly offered to 
care for them at the end 

D. They were all good at doing tricks. — (OOB) though they were intended to help her, it 
was not by doing tricks 
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This analysis represents the performance of: 



School Sample ES Grade: 04 Class/Sec: 08 



Question Set: G4F1 



4. What is the MAI N I DEA of this story? 



A. Sometimes you can get what you want by helping others. — Sam eventually gets to take 
care of the pets that he wants by helping Mrs. Cabot solve her various problems 

B. Sometimes you can find pets at an animal shelter. — (OOPl) the cat was, but this is not 
the main idea of the story 

C. Sometimes there's not enough work to do around an apartment building. — (00P2) 
there was too much work to be done, and it's not the main idea 

D. Sometimes you can meet new people near your home. — (OOB) this may sound 
reasonable, but is not related to the main idea of the story 



5. According to the passage, why did Sam think Mrs. Cabot should get a goat? 

A. to chase away robbers — (00P2) he suggested a dog to chase robbers 

B. so she could start a petting zoo next to the apartment building — (OOB) goats are 
common in petting zoos, but this has little basis in the text 

C. so she wouldn't have to mow the lawn — found seven paragraphs from the end of the 
passage 

D. to cure her allergies — (OOPl) the goat won't "cure" her grass allergies, but would help 
alleviate them by reducing her need to mow the lawn 



6. What was the MAI N reason Sam kept suggesting animals for Mrs. Cabot to get? 

A. He wanted Mrs. Cabot to have a pet instead of him. — (00P2) Sam hoped he AND Mrs. 
Cabot could have pets. 

B. He liked helping Mrs. Cabot. — (OOPl) Sam offered to mow the lawn and pet-sit, but 
helpfulness is not the main reason he wants her to get pets 

C. He hoped Mrs. Cabot would let him get a pet too. 

D. He wished that Mrs. Cabot would move away. — (OOB) 



7. Who did Sam think would be a good pet-sitter for Mrs. Cabot? 

y A. himself — can be inferred from Sam's final statement in the passage 



B. his parents — (OOPl) Sam's parents "even live in the building" and seemed to support 
his desire for a pet, but there's no evidence the want to care for pets 

C. Mr. Cabot — (00P2) Mr. Cabot "even lives in the building" but the paragraph describing 
what he likes to do does not include caring for animals 

D. the man who walks his dog past the building — (OOB) 



4. Making Inferences 
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This analysis represents the performance of: 



School Sample ES Grade: 04 Class/Sec: 08 



Question Set: G4F1 



8. Why did Sam grin when he suggested Mrs. Cabot buy a cat? 



8. Making Inferences 



A. He thought it was funny that Mrs. Cabot hated mice. — (OOB) a grin may signal humor, 
but Sam "rushed over" out of concern when she screamed 

B. He hoped Mrs. Cabot would get a cat in spite of her rule against pets. — can be inferred 
from his questions to Mrs. Cabot about getting a pet 

C. He liked mice.— (OOPl) Probably true, since he wanted "any kind of pet at all," but not 
the reason he grinned 

D. He knew that Mrs. Cabot liked cats. — (00P2) Sam himself liked cats, but Mrs. Cabot 
"hated cats" 



9. The MOST LI KELY reason that Mrs. Cabot did not allow pets was that 

A. she didn't like animals. — can be inferred from the fact that each time Sam suggests an 
animal to solve her problem, it says Mrs. Cabot "hated" the animal 

B. she didn't want too many animals in the building. — (OOPl) at the start sit says he didn't 
want ANY animals in her building 

C. she wanted to run a zoo. — (00P2) Mrs. Cabot snapped, "Then I 'd be running a zoo and 
not an apartment building," implying disfavor at the idea 

D. Mr. Cabot was afraid of animals. — (OOB) no evidence suggests this 



10. Why did Mrs. Cabot want to sell the apartment building? 

A. She did not want to take care of all her pets anymore. — (00P2) she's too busy to care 
for animals and the building, but seems to be keeping the animals 

B. She was afraid of robbers. — (OOPl) she was afraid of robbers, but that's why she got a 
dog, not why she wants to sell the building 

C. She thought she could make money from selling the building. — (OOB) logical (especially 
in Boston's real estate market!) but not implied in the passage 

D. She could not take care of both the animals and the apartment building.— found four 
paragraphs from the end of the passage 
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Question Set: G4F1 

School: Sample ES 

T otal Students: 20 

Stud I D # First Name Last Name Q1 Q2 Q3 Q4 Q5 



Student- Level Data Summary 



Key to colors: 



Skill Summaries: 

FE # Ml # FE % 

Q6 Q7 Q8 Q9 QIO 
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"mis- read" 
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Finding Making 



Types and Levels of Thinking Assessed on FAST-R 

Formative Assessments of Student Thinking in Reading 



FAST-R assesses two primary ways that readers construct understanding as they read: finding evidence to deter- 
mine explicit meaning in the text, and making inferences to determine implicit meaning in the text. 



The graphic below represents how these skills build on one another as readers answer questions of increasing 
sophistication. FAST-R Teacher Guides annotate each question with a miniature version of this graphic to highlight 
the level of thinking that question assesses. 
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“Making inference” questions ask readers to 
determine implicit meaning ... 
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... from the total text. 






... from particular words, phrases, or ideas in 
context. 



... by 

incorporat- 
ing literary 
knowledge. 




^ “Finding evidence” 


... by recognizing evidence explicitly stated at multiple locations 


1 questions ask readers 


or with varied wording in the text. 


■ to determine 




^ explicit 
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... by identifying evidence explicitly stated at one location in the text. 



“Finding Evidence” Questions 

Readers find and use evidence to determine the author’s explicit meaning from details that are stated in the text. 
There are two main types of “finding evidence” questions on FAST-R. 



Identify evidence explicitly stated at one location 
in the text 

This type of referential question asks readers to iden- 
tify and locate an answer explicitly stated somewhere 
in the text itself (including the introductory blurb, 
author spotlight, and footnotes). To answer this type 
of question, students must look back at the passage 
and “identify” the answer as it is stated in the text. 



Recognize evidence explicitly stated at multiple 
locations or with varied wording in the text 

Although answers for the second type of “finding 
evidence” question are also explicitly stated in the 
text, readers may need to gather the relevant details 
from several places in the text or recognize words and 
phrases that express the same meaning in slightly dif- 
ferent ways. Students are not asked to make any new 
meanings or draw any new conclusions (as they do to 
answer inferential questions). 
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“Making Inference” Questions 

Readers make inferences when they determine implicit meaning that is implied but not directly stated. FAST-R 
“making inference” questions ask readers to fill gaps based on knowledge gleaned from a close reading of the text. This 
close reading can help students avoid basing inferences on incorrect or irrelevant prior knowledge. 



Type 1: Determine implicit meaning from words, 
phrases, or ideas in context 

In this type of inferential question, the word, phrase, 
or idea to be interpreted is identified in the ques- 
tion stem itself. However, the question does not tell 
students what other clues to use. In other words, 
students may be directed to a starting place, but they 
may need to employ other close reading strategies and 
base their interpretation on additional textual clues. 

For example: Vocabulary questions, cause-and-effect 
questions, and some basic “author’s craft” questions. 

• “Based on the clues in paragraph 6, what is the best 
definition of the word troupeT' {word in context) 

• “In paragraph 13, the author describes a rainbow as 
‘an enormous arch of colored stripes,’ suggesting ...” 
{phrase in context) 

• “In paragraph 9, why did the Admiral ‘make obser- 
vations from different points’?” {idea in context) 

To answer these questions correctly, students must 
draw conclusions and derive implied meanings from 
several places in the text. 

Type 2: Determine a single implicit meaning from 
the total text 

This type of inferential question asks students to draw 
a single, overarching new meaning from the entire 
passage (or from entire paragraphs or sections). Infer- 
ring in this way involves multiple thought processes: 
Students must comprehend most or all of the details 
in the text, then synthesize the information in order 
to understand the larger meaning. 

For example: “Main idea” and “main purpose” ques- 
tions asking about some chunk of the text. 

• “In paragraph six, the author’s main point is ...” 

• “Which sentence best expresses the main idea of this 
excerpt?” 

Type 3: Determine implicit meaning by understand- 
ing the organization of information in a text 

This type of inferential question requires readers not 
only to derive new meanings, but also to analyze. 



order, or prioritize those ideas in light of the way the 
text is organized. This process involves “analyzing” and 
“evaluating” (as described in Bloom’s Taxonomy), and is 
used as readers “determine importance” and distinguish 
supporting details from larger ideas. 

For example: These questions often use words like “best” 
or “most important.” 

• “What would be the best heading for the information 
in paragraph 9?” 

• “Which detail best supports the idea that the narrator 
has adjusted to life in America?” 

Type 4: Determine implicit meaning and apply it be- 
yond the passage context 

This type of inferential question requires readers to deter- 
mine a new meaning from the text and apply that mean- 
ing in a situation that extends beyond the text itself 

For example: 

• “Based on the details in the passage, if Martin were 
choosing another book to read, he would most likely 
choose ...” 

• “Which topic will the author most likely write about 
in the chapter immediately following this prologue?” 

This thinking process often involves identifying relevant 
details in the text, accurately interpreting their implied 
meaning, and then matching or classifying that idea to 
make sense of a new situation. 

Type 5: Incorporate literary knowledge to determine 
implicit meaning 

These inferential questions require students to incor- 
porate their knowledge of literary terms or concepts in 
order to understand the text. Depending on what kind 
of interpretation the question asks of readers, the ques- 
tion may also exhibit characteristics of Type 1-4 
“making inference” questions. 

For example: 

• “Why did the author end paragraph 5 with ‘- and 
disaster struck’?” 

• “Which of the following details helps create a tone of 
anticipation in the passage?” 
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